Hi Jonathan Thanks for the follow up.Also - apologies as I realised I used my non-linux friendly email account for my original posting with all the horrible outlook mangling. I'm posting this from my alternative account which avoids outlook (I'm just transitioning between the two accounts...)
On 6/3/2020 1:23 PM, Mark Pearson wrote:
-----Original Message----- From: Jonathan Carter <jcc@debian.org> Sent: Wednesday, June 3, 2020 11:54 AM Hi Mark! On 2020/06/03 15:39, Mark Pearson wrote:I guess it should be possible, but the biggest challenge is probably to get the right contact to talk with. Do we have such a contact ?I started replying to this thread late last night and after a few iterations Igave up and went to bed 😊 Here goes attempt 5.... First of all, thanks for the great e-mail and making the time to reach out. By way of introduction, I'm the current Debian Project Leader (since a bit more than a month now) and I am happy to be available to help speed things up, feel free to reach out to me personally at any time if you're frustrated with getting something done in Debian.
Congrats on the DPL - I did follow that process :)
Short answer - you have the right contact. I'm the linux technical lead atLenovo for the PC team and I'd *love* to improve the Debian experience on Lenovo platforms. It's something I have actively been trying to work on for the last 9 months (with limited success). I'm really happy to see this question and the responses - I would really like to have this conversation with the wider community as to what Lenovo and Debian can do to work better together. Debian is an important distro with a lot of users and an amazing community. I was hoping to be at Debconf 2020 and use that as an opportunity to actually get to meet Debian devs as frankly email has been working poorly and maintainers are just very busy people. Sadly I think Covid has likely stomped on that plan. Talking at DC20 would've been great, but we can also talk via email and video calls in the meantime!
I'd be totally up for that.
Awesome - I just watched the video and you make some great points and I think there are a bunch of questions in there that I can answer - or that we're actively working on.So and let me start the ball rolling by highlighting some good and bad: The good - Lenovo are expanding what they offer on Linux (we had anotherbig announcement yesterday about doing full config on our workstations with Ubuntu and RHEL). We're asking HW vendors to have Linux support upstream including firmware on LVFS which I think is important. It's not perfect yet but it's getting better and better. We're starting to contribute to open source projects.I'm an open source user and a lot of what I see from Lenovo internally ispositive and makes me very happy to advocate for what they are doing. Sometimes our process is a bit slow but there is a commitment to doing Linux and doing it right which I'm personally excited by. The Fedora collaboration has worked really well - I think for both sides (obviously talk to your Fedora colleagues for their perspective). They have been a really positive community to work with and it's been productive showing that Lenovo can collaborate with an OS community in the right way.Internally at Lenovo there is still a lot of learning about Linux and how itworks - and that is happening. Linux is catching on - I have more customer engagements and our Linux sales have been increasing and we haven't turned our websales on yet (coming soon 😉)The bad - I love Debian but I don't think as a distro you are ready or capablefor being pre-loaded on our platforms. I can explain why and I'm very happy to be corrected but more importantly if it can be solved then we can for sure explore the next steps - that would make me really happy! I 100% agree with you, in my first talk as DPL, this is even a topic that I've covered when talking with the Brazillian Debian community: https://peertube.debian.social/videos/watch/cdfe5b24-e3ad-4422-a6f2- 15ca4fffd895?start=43m52s (I talk about this for about 5 minutes, and it was almost midnight and I was up since 5am so sorry for maybe being a bit incoherent at times)
So - I'm kinda nervous about the support side to see what happens as we enable websales. Up until now it's just been corporate customers and best effort support on our Lenovo Linux forums. We will provide support for HW related aspects of the platforms and genuinely we're waiting to see how bad or good it's going to be. For OS stuff that isn't HW related (e.g. the calendar app crashes) then we'll point more at the community. My hope is that Linux users are more technically aware and will be reasonable....We will see what happens but it is a bit of an unknown. I'm spending a fair chunk of time at the moment following up with the Lenovo support team getting them familiar with the basics of Linux.Anyway, let me summarize what I said on the topic in that video with some additional notes. Firstly, I think the typical Debian developer might not care about Debian being available on an OEM laptop per sé. First thing that most of us would probably do is to re-install it from official installation media :) But there's probably a vast amount of Debian users out there that would want a Debian on their ThinkPad pre-configured. And we'd be delighted from the Debian side if we could help make that happen. So in the video I posted some of the initial thoughts that I think we need to sort out in Debian before we can even start thinking about this. I'm glad you reached out because you'll be able to give us much better insight for sure, but at least I have some video evidence to prove that we put at least *some* thought in to it :) 1. Support I'm not sure what you did in the case with Fedora, but my guess is you have some agreement with how to handle support calls for Fedora on the system. Debian isn't backed by any one company, and in the immediate future our usual community support channels might be as good as it gets. I suppose if users know this, understand this, and know what they're getting, then this becomes less of an issue.
2. OEM Mode installer Currently we don't have an OEM mode install mode in Debian (that is, skip questions like timezone, user, password, etc and ask that on first boot instead). I believe that there's been some work to make a debian-installer module for this, but it's been shelved since. Either way, this isn't a difficult problem to solve on a technical level, and if we need it to get Debian available on laptops then I think it would happen fast.
That's a good point - we do have that on Fedora.
Actually I have an ongoing exercise to improve the recovery side of things with a meeting later this afternoon.3. Rescue partition Laptop manufacturers usually don't ship with physical media anymore. Instead, the laptops have a rescue partition on them for re-installing/resetting the machine. As far as I know both installers we currently use in Debian are fine from installing from a rescue partition, we just need a nice way to set that up when initially performing an oem style setup from our installation media. (again, not a huge technical problem, but probably a bit more work than #2).
For Fedora given what we are providing is just Fedora + some docs then the recovery solution is currently "go install Fedora using their USB installer and the docs are on the support site if you really want them". It's not the best but it's what we have.
...and then there's hardware specific stuff...Debian moves somewhat slowly. The general approach seems to me to be towait for updates to trickle down from the upstream community and I completely understand that approach - I'm sure it is important to maintaining stability *but* it means that Debian is usually broken on our platforms for the first year they are available because that whole process takes a long time. By the time Debian works it's too late for a pre-loaded offering. As an aside RHEL has the same problem but they put a lot of effort into backporting fixes so that their customers get the fixes - and that infrastructure doesn't seem available on Debian to my limited knowledge. We have processes for both stable release updates and backporting too. Where they fall short, I'm sure we can look into it further.As an important example - the X1 Carbon 7 (which is a popular machine) stilldoesn't work well with any version of Debian (including experimental or testing) as the audio is broken. Debian users have to jump through a few hoops to get it to work. I've let the maintainer know a number of times what is involved to fix that but it's obviously not a priority (as a heads up - Debian on most Lenovo 2020 platforms is going to suck because of this too). I'm not meaning to point fingers - but just explain why it feels as if Debian and the latest hardware is an awkward fit. I think what we need to figure out from the Debian side is, whether it's possible to integrate these kind of 'hot fixes' using our usual procedures and processes, or whether we might need a lenovo-specific image and even perhaps its own package archive. We'd of course like to avoid that, but just from me personally, I think it's worth exploring what it would take to make it work. If you're willing to share what worked with RHEL, then maybe we can adapt some of that to our processes too.
Yep - that would be good.Ubuntu has the concept of oem image which covers the period between new HW availability and it making into the standard image but they also have the benefit of a lot of resources working on it.
With Fedora they have the RH team supporting them which was how we became more involved with the community - we work with RHEL and have done so for a while.
We raise bugs the normal way with Fedora. They weren't public before the announcement but we're moving to doing things in a more standard public way. We have the advantage that a bug on Fedora is usually something RHEL wants so it's easy to get attention and that would be something Debian would need to consider. We did get input for their bug triaging but ultimately it was for the community to decide what was a blocker or not and I didn't win every battle. There was an audio over HDMI nouveau driver issue that got fixed after F32 was released - users will get that just as part of their regular update and that's OK.
In my mind the best long term solution for Debian is if Lenovo are contributing fixes ourselves - after all we usually know what the fixes are. We're not technically at the stage where that is doable though - but it's what I'm aiming for and I'm the trial guinea-pig. For me the first step would be an understanding that if we propose a patch then we're doing it for a good reason and it doesn't sit there for a couple of months. I don't think we're at the stage where you can trust us to merge things in without review but longer term I'd hope to be a more trusted community member who can be relied on to deliver quality items. How we get from here to there is in my mind the challenge - for that we need support and I appreciate peoples time is limited and very precious :)
From my point of view what I've been trying to do is to get more involved soI can contribute/backport fixes directly. I get good insight into what issues impact our platforms and when fixes land upstream. It seems the best way to make contribute and make a difference. Unfortunately I've still got a *lot* of learning to do and it's a really slow process because the loop between offering a fix, getting it reviewed to find out what you did wrong and contributing the update is crazy slow (for example I have kernel MR 240 open for four weeks for an OLED brightness issue that a lot of users think is important). My expectation is that as I make fewer mistakes and earn some trust that will help - but until that point (which I'm guessing will take years 😊) I have limited handles that I can pull on to make things happen. You're right that the trust factor is important in Debian. It helps when there's a Debian Developer or two who can guide you through some of the murkier aspects of the project. Again, if I can help you with any of that, I'd be happy to. I don't think it needs to take years though :)
That sounds good.
Agreed. I'm afraid from a Lenovo preload shipping point of view it has to be the latest HW though and that's a challenge. If it's any consolation, whilst I have access to a lot of devices I also have to fight for some HW access. I don't have an X1C8 yet (my colleague does....I'm not bitter ;)) Getting HW where it needs to be is one of the big challenges I face generally.My *impression* is there is limited desire to accelerate fixes for Lenovoplatforms - I suspect mostly because people are just plenty busy with the things they care about instead and I understand that. I think we generally care about the hardware that we have, and that our employers typically buy and deploy. I think "other people's hardware problems" will always be less exciting than your own. Usually when more of the latest hardware starts hitting us personally we tend to care more. I think part of this is just natural.
I've kicked off a discussion on this internally. I did actually look into this when I joined the team last year but I know more the right people to talk to now and Linux is getting a lot more attention internally. I'll see what I can do - I think it's a very reasonable suggestion.Maybe since I have your attention I can share some of my thoughts too that might be directly relevant to the above. We have many ThinkPad users in Debian, a very large percentage of Debian Developers have owned at least one ThinkPad (and some multiple ones over the years, and some multiple ones even at the same time!)). The work that these Debian Developers do typically go towards helping those Lenovo laptops work better with Debian. I understand and feel your frustrations getting those latest features to work in Debian, but Debian people have been doing that for decades now, which is great, I'm not complaining. I like that Debian has run so well on my ThinkPad laptops for all these years. But, in recent years I've also started to think that it's kind of horrible and ridiculous that Debian Developers pay full retail price for their laptops considering how much work they put in making Debian (and often other Linux systems) run better on it. I've been considering reaching out to both Lenovo and Dell (and other manufacturers that are popular or important within the project) to see if we can get much better pricing for our members, whether they're just closer to cost or even a little subsidized. Maybe you could shed some light on this topic too :)
On another note, we sometimes use older laptops not because we're necessarily stuck with them, but also because we don't want to throw away/discard perfectly good machines just for the sake of having a new one, while we could use our older machines and help do a bit to protect the environment at the same time. My x250 is 5 years old and it's still adequate for me, and while I often have the itch to buy a new laptop (I very nearly did buy a friend's 7th gen X1 last week, which he bought sadly just before he lost his job), but decided to hang on for a Ryzen based version instead. Sorry, I digress :)
I hear you :)
So, we have some people (like me) who use 5 year old (and even older) laptops and are fine with that, but we also have some developers who are using really obsolete machines and from the Debian side, I'd like to make it easier for them to upgrade if we can. Debian can already sponsor some hardware for developers in need, but personally my vision is that we can work with a hardware manufacturer to get an already good/subsidised price that we could perhaps help reduce a bit further as a benefit for our developers. Over the last weekend we had our first ever online minidebconf, and in these times being stuck on a core 2 duo is tough : >> Anyway, I probably overshare but you've put in a lot of thought into your mail and I thought I'd reply properly too.A shout out for Hector Martinez who has been helping me a whole bunch.Without his help I likely would have given up and wouldn't even be reading the threads on this forum. If there are more people like Hector (particularly in kernel, audio and graphics) let me know! As I become more aware of your requirements I can help find more Hectors too :)This email took me a long time to write - I'm *very* aware that I'm new tothis and don't want to cause offence. Please take all of the above with the recognition that my viewpoint into Debian is still limited and if I've said anything dumb/wrong/offensive let me know so I can learn what I'm missing. No offense taken whatsoever, Debian Developers are often the biggest critics of Debian. Not because they want to be mean, but because they care and want to see things get fixed. Welcome to Debian and I hope that you'll join us more formally (and perhaps even vice-versa?) in the future!
Thanks for the follow up - it is appreciated.
-Jonathan -- ⢀⣴⠾⠻⢶⣦⠀ Jonathan Carter (highvoltage) <jcc> ⣾⠁⢠⠒⠀⣿⡁ https://wiki.debian.org/highvoltage ⢿⡄⠘⠷⠚⠋ https://debian.org | https://jonathancarter.org ⠈⠳⣄⠀⠀⠀⠀ Debian, the universal operating system.