24.04 also ships with a footgun that keeps PasswordAuthentication enabled even if you edit /etc/ssh/sshd_config. It adds a /etc/ssh/sshd_config.d/50-cloud-init.conf that force overrides any PasswordAuthentication settings you have configured in /etc/ssh/sshd_config.
A better practice is to use the `.d/` convention as intended - so you can have overrides and customizations without having to edit the OS-managed /etc/ssh/sshd_config file and fight it if an OS upgrade changes it.
Edit: It's not really a mistake on Ubuntu's part, and is common in other distros for overriding upstream defaults[1].
The proposed solution is bad, the file will be restored by the package manager at some point. Always prefix your override files with '99-' or 'zz-', this would have prevented the problem.
For OpenSSH configuration the first directive controls, not the last. So if you want to set an option make sure your include filename comes lexically earlier, not later. This is why the Include directive is at the top of distro-installed /etc/ssh/sshd_config. You can verify your server configuration using `sshd -T`, and a client configuration using `ssh -G`; they dump the resolved configuration to stdout.
I don't have /etc/ssh/sshd_config.d/50-cloud-init.conf on my ubuntu 24.04 machines. What creates it? Is it a clean-install vs. upgrade difference perhaps?
This file is created when the server is provisioned with cloud-init. For example when you use https://cloud-images.ubuntu.com/ as a template image to create your VM. For these cases you would need to supply cloud-init config via separate means (openstack user-data or just secondary mounted ISO) and cloud-init would perform post-install tasks like configuring users, passwords, ssh keys, etc. If you just install your machine from ISO, cloud-init is not used.
I guess that many hosters will use cloud-init for their VPS offerings "under-the-hood". Usually they'll generate password and mail it to you, so obviously ssh password should be allowed for this case.
Unfortunately, one possibility is the user chose "enable SSH password authentication" during the install [1]. Or it's a cloud instance from a provider that provisions instances with a password.
Great, now you can start putting your customizations in that directory instead of the OS-managed /etc/ssh/sshd_config blob. That's why the `.d/` convention exists.
I had to look up the info because I wasn't sure really, so I appreciate the direct question. It means 'directory' or 'conf.d pattern'. So you have your config file, and the config directory that contains "parts" of the config to be merged with the main one to provide customizations/overrides.
Err, why? Nothing is actually broken with Ubuntu 24.04. The issue GP is describing is just a lack of understanding of how config files have worked on Linux for decades.
On top of that, this issue doesn't even "ship with 24.04". GP probably chose to enable password auth in SSH during installation, or they used a cloud provider that provisions instances with passwords and overrides the default.
Actually ubuntu and/or some cloud integrator is violating principle of least suprise, which predates by even more decades and applies everywhere not just in a computer operating system.
Having /etc/ssh/sshd_config not do what it appears to do is inexcusable, however that comes about.
Even if the fix is no more than structuring the file a little differently so there are parts both before and after the .d loading, and comments that say what overrides what, and all the actual config files and default behavior are still the same.
That's just not how it works, and it never has. One of the main purposes of the .d files is to avoid modifying the distro's sshd_config file, so you don't have to resolve conflicts during an OS upgrade. Commenting your overrides in the sshd_config? That's backwards.
Downstream distros sometimes want to override the default settings. That's what .d is for. It doesn't violate the dogma "principle of least surprise", especially if you run `man sshd_config` and learn the first thing about sshd's config files.
Any sufficiently poorly constructed system is indistinguishable from a legitimate vulnerability.
You shouldn't need to edit an undocumented file that's force enabling an insecure authentication method in the first place.
"You're using it wrong" doesn't change the fact that the defaults are poor and likely to ultimately cause more devices to be improperly configured than if it was left as sysadmins and end-users expect.
They likely chose the option to enable password-based SSH authentication during installation [1], or it's a cloud service that provisions servers with a password.
I've been looking for info about whether it's force-enabled with a vanilla install where the user doesn't actually choose for it to be enabled. I'm happy to be educated here.
There is nothing broken with Ubuntu, just people not understanding how configuration files work in Linux, choosing to enable password auth in SSH during installation, or using a cloud provider that provisions instances with passwords and overrides the default.
You can still configure it to be key only, you just need to put your own override as a file in /etc/ssh/sshd_config.d/ rather than /etc/ssh/sshd_config. The files are read ins order, so your filename needs to sort after the 50-cloud-init.conf file.
This would work:
echo "PasswordAuthentication no" > /etc/ssh/sshd_config.d/60-password-auth.conf
To be precise: you don't need to be in the sudo group, but in the lpadmin group. I'm not familiar with how Ubuntu groups are set up, but I guess it's likely that lpadmin is only granted to administrators by default.
That said, I'm guessing people aren't expecting lpadmin to mean a full privilege escalation to root.
There are two bugs here: one in cups, which allows it to chmod anything 777 (doesn't properly check for symlinks, or for the failure of bind), and one in wpa_supplicant, which lets it load arbitrary .so files as root. However, I suspect that even if these bugs are fixed, having access to lpadmin will still be a powerful enough primitive to escalate to root given the rather sizable attack surface of cups.
It became crystal clear that cups is a can of worms, and it would be prudent to completely replace it with with a new solution built from the ground up, ideally using modern tools and standards.
or just sandbox cups. There's no reason cups needs to write anything beyond its "configuration" and its "print spool". And hence, shouldn't have access to anything beyond what it needs to configure itself and print.
thing like cups should be easy to sandbox, especially if we allow dbus like APIs as a means to cross sandbox boundaries (i.e. RPC mechanism).
and by sandbox, I dont mean simply use apparmor type rules (though that can work), but a cups that lives within its own file system and nothing else is even visible.
i.e. programs will always be buggy, even if we get rid of all language oriented bugs, there will still be logic bugs that will result in security holes. We just need to make it easy to isolate programs (and services) into their own sandboxes while retaining the ability for them to interact (as otherwise, lose much of the value of modern systems).
In practice, I would argue, a lot of modern systems do this already (ala ios/android). The apps run sandboxed and only have restricted abilities to interact with each other.
That's sort-of the direction they're going with CUPS 3. The 'local server', which is what most people will need, runs as a normal user, not root, doesn't listen on the internet, and talks only the IPP everywhere protocol. For supporting legacy printers, there will be separate sandboxed 'printer applications' which read in IPP Everywhere, run the driver code, and communicate with the backend printer using an appropriate protocol.
For enterprise users there will be a separate 'sharing server'.
I'd prefer they make it the default to not install it. I don't need to print from Linux. I don't print from Windows nor MacOS much either. Less than once a year. But I particularly don't print from Linux. I suspect that true for most people. cups shouldn't be a default install IMO
How many percent is "most" people? What about enterprise users with complex setups/requirements, will they be supported or out-of-luck? Typically you'll have print servers with centralized authentication, possibly logging/auditing/billing, any this might depend on "the" component they'll leave out in the new product because, well, most people don't care about it...
> How many percent is "most" people? What about enterprise users with complex setups/requirements, will they be supported or out-of-luck? Typically you'll have print servers with centralized authentication, possibly logging/auditing/billing, any this might depend on "the" component they'll leave out in the new product because, well, most people don't care about it...
But the old, complex cups doesn't go away if a new, sandboxed version is developed, so the people who want the complexities can evaluate whether the security trade-off is worth it, and use it anyway if so.
What if you don't need cups because you don't print anything?
Just sudo apt remove cups right?
No, because cups is a dependency of the entire graphical subsystem, just removing cups also removes everything from the Nautilus file manager to Firefox to ubuntu-desktop itself.
To expand on this: if the user is in the sudo group, they have explicit permission to execute anything they like as root. If someone wants a user to not be able to do this, they don't put that user in the sudo group. As far as I can tell from the write-up, if you remove a user from the sudo group because you don't want them to have that privilege then this "exploit" won't work.
The bugs found look correct and have security implications, but what is demonstrated is therefore not really "root privilege escalation" since it applies only to users who already have that privilege.
> They can execute anything they like as root... by entering their password.
If it has control of your user account, then it can just arrange to wrap your shell prompt and wait for you to sudo something else. The sudo password prompt in its default arrangement doesn't really provide much security there and isn't expected to.
On a properly configured server you'll be waiting forever, because the users actually running the applications on that server aren't the same users who have privileges to make changes to the system or have access to stuff like sudo. So if you take over the nginx/postgres/whatever user, you're not really going to get anywhere.
On the other hand you probably don't need to. Those users already expose all the juicy data on the server. You don't gain much from obtaining root anyways, except better persistence.
This attack might be more interesting when chained with some other exploit that gains access to a users system via their e-mail client or browser. In other words nice if you're NSO Group making exploits for targeting individuals, but not that useful if you're trying to make a botnet.
That's not really relevant nowadays. Most attacks are done indiscriminately and en-masse, so an attacker wouldn't have to wait very long in practice.
Only in "advanced persistent thread" territory is your point really relevant, but the attack I describe is much more widely applicable. Having to wait a while is therefore not in any way a mitigation. In practice then, one cannot assume any security from sudo requiring a password.
Somewhat tangentially, I will say that Touch ID-based sudo is a real upgrade over password sudo. It still gives you that extra moment to reflect on whether you really want to run that command (unlike passwordless sudo), without being burdensome.
If that is your attitude, why bother with the sudo group at all? Just run as root.
(For what it's worth, I think most people would not lose much security from running as root, and the obsession with sudo is so much security theater, for exactly this sort of reason.)
User accounts and sudo does auditing of who is doing what. They’re are other ways, sure, but checking auth.log is the simplest.
And while lpadmin users can escalate, I’m more interested in escalations from services like web servers or whatever, running as low priv users. I use sudo to allow scripts running as my web server to run specific limited privileged programs as a simple layer of defence.
Honestly, sudo’s value is really sanity, not security.
The first time you use certain flavors of sudo, you get a nice little message which reminds you why sudo exists:
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
Realistically, sudo exists to remind a user of these points. That is: by needing to type “sudo” before a command, you’re being reminded to pay closer attention that you’re not violating another user’s privacy or doing something that’s going to break your system.
Sudo is so commonly used especially on developer machines that I think it is used reflexively without any thought at at all.
It should not be, but that's a different issue. It amazes me the amount of open-source projects that want to be installed with "sudo" when there is no reason they should not be able to be built and used entirely from within the developer's home directory.
I know more than one person who starts a shell session with "sudo -i" and then just works as root because typing "sudo" all the time is an annoyance.
I wonder if this comes from the how some developers view ops knowledge and tasks as merely ancillary to their interests and work.
For me, Linux was a hobby prior to and separately from programming. In the tutorials and documentation I read, every command was explained in detail and it was emphasized to me that I should never run a command that I don't fully understand. All instructions to elevate privileges were accompanied with advice about being careful running commands as root because root's privileges are especially dangerous. I was interested in those warnings, and took them seriously, because I wanted to master what I was learning. What I was learning, though, explicitly included Linux/Unix norms like security 'best practices'.
Developer documentation doesn't usually concern itself with Linux/Unix norms the way that tutorials for Linux hobbyists and novice sysadmins do. At the same time, the developers reading it might be perfectly dedicated to mastery, but just not really see what is considered proper usage by sysadmins (let alone the considerations that inform the boundaries of such propriety) to on-topic for what they're studying/exploring/playing with. Diving into those details might not be 'patt of the fun' for them.
What such a developer learns about sudo is mostly going to come from shallow pattern recognition: sudo is a button to slap when something doesn't work the first time, and maybe it has something to do with permissions.
But I think that comes from the mode of engagement, especially at the time of learning sudo, more than the mere frequency of use. I use sudo several times every day (including sometimes interactive sessions like you mention, with -i or -s), but I am careful to limit my usage to cases where it's really required. I'm not perfect about that; occasionally I run `sudo du` when `du` would suffice because I pulled something out of my shell history when I happened to be running it from / or whatever. But I certainly don't run it reflexively or thoughtlessly.
Use sudo instead of root. Change ssh port. Disable ssh passwords. Use some port knocking schemes or fail2ban. Close all ports with firewall. Some of those requirements might come from some stupid complicance rules, but usually they just come from rumors and lack of clear understanding of threat model. And sometimes they might even slightly reduce security gurantees (like changing ssh port to 10022, there's actually a security reason why only root can bind to <1024 port and by changing it to higher port you lose some tiny protection that unix greybeards invented for you).
I'm not saying that all those measures are useless, in fact I often change ssh port myself. But I'm doing that purely to reduce log spam, not because that's necessary for security. Configuring firewall might be necessary in rare cases when brain-dead software must listen on 0.0.0.0 but must not be available outside. But that's not something given and should be decided in case-by-case basis, rather than applied blindly.
That was a great read. The way the author builds the exploit, brick by brick, is well done and not all all obvious or clear. Each step by itself is somewhat concerning but there's no Eureka! moment until very late.
I wonder if there's a tool to create dependency graphs for these dbus and polkit interactions, ideally to better audit those that seem to cross interesting trust boundaries.
The only feedback I get when installing d-spy is "Uses System Services", and "Uses Session Services", which means nothing to me as a user, and yet it allows program to enumerate all programs I use and as it turns out even hack my computer. Other platforms solved this with something like "developer mode", iOS, Android, Meta, etc. I shouldn't be able to install this app without confirming developer-mode-only permissions. As for this particular app it is offline, yes, but dbus allows for cross-app communication, so no more
You’re running desktop Linux - your whole system is permanently in “developer mode”. That’s kind of the point? Windows and macOS don’t have “developer mode” either, at least not to the extent seen in the mobile OSes.
See here: https://news.ycombinator.com/item?id=42133181
Edit: It's not really a mistake on Ubuntu's part, and is common in other distros for overriding upstream defaults[1].
[1]: https://askubuntu.com/a/1516347
I guess that many hosters will use cloud-init for their VPS offerings "under-the-hood". Usually they'll generate password and mail it to you, so obviously ssh password should be allowed for this case.
[1]: https://askubuntu.com/a/1440509
I thought I fixed it, but apparently not. It is driving me crazy.
On top of that, this issue doesn't even "ship with 24.04". GP probably chose to enable password auth in SSH during installation, or they used a cloud provider that provisions instances with passwords and overrides the default.
Having /etc/ssh/sshd_config not do what it appears to do is inexcusable, however that comes about.
Even if the fix is no more than structuring the file a little differently so there are parts both before and after the .d loading, and comments that say what overrides what, and all the actual config files and default behavior are still the same.
Downstream distros sometimes want to override the default settings. That's what .d is for. It doesn't violate the dogma "principle of least surprise", especially if you run `man sshd_config` and learn the first thing about sshd's config files.
"That's just not how it works" Incorrect.
"dogma "principle of least surprise"" That is not dogma, it is a principle. It's right there in the name.
Dogma might be the understanding that the principle of least suprise is a valid principle.
But this suggests you don't subscribe to this silly "dogma" eh?
You shouldn't need to edit an undocumented file that's force enabling an insecure authentication method in the first place.
"You're using it wrong" doesn't change the fact that the defaults are poor and likely to ultimately cause more devices to be improperly configured than if it was left as sysadmins and end-users expect.
I've been looking for info about whether it's force-enabled with a vanilla install where the user doesn't actually choose for it to be enabled. I'm happy to be educated here.
[1]: https://askubuntu.com/a/1440509
Was thinking of upgrading but not if I can’t configure SSH to be key only
This would work: echo "PasswordAuthentication no" > /etc/ssh/sshd_config.d/60-password-auth.conf
Had to check my other VMs. Luckily most of them are debian or 22.04 (for now).
Great read, but this feels like academic research. Technically correct, but impractical at best.
That said, I'm guessing people aren't expecting lpadmin to mean a full privilege escalation to root.
There are two bugs here: one in cups, which allows it to chmod anything 777 (doesn't properly check for symlinks, or for the failure of bind), and one in wpa_supplicant, which lets it load arbitrary .so files as root. However, I suspect that even if these bugs are fixed, having access to lpadmin will still be a powerful enough primitive to escalate to root given the rather sizable attack surface of cups.
thing like cups should be easy to sandbox, especially if we allow dbus like APIs as a means to cross sandbox boundaries (i.e. RPC mechanism).
and by sandbox, I dont mean simply use apparmor type rules (though that can work), but a cups that lives within its own file system and nothing else is even visible.
i.e. programs will always be buggy, even if we get rid of all language oriented bugs, there will still be logic bugs that will result in security holes. We just need to make it easy to isolate programs (and services) into their own sandboxes while retaining the ability for them to interact (as otherwise, lose much of the value of modern systems).
In practice, I would argue, a lot of modern systems do this already (ala ios/android). The apps run sandboxed and only have restricted abilities to interact with each other.
For enterprise users there will be a separate 'sharing server'.
https://ftp.pwg.org/pub/pwg/liaison/openprinting/presentatio...
Many Linux users and developers don't run anything else. If they're to print at all it'll be from Linux.
Which sounds fine? Most people don't want LPT printers support, they want AirPrint and WSD to just work.
But the old, complex cups doesn't go away if a new, sandboxed version is developed, so the people who want the complexities can evaluate whether the security trade-off is worth it, and use it anyway if so.
Just sudo apt remove cups right?
No, because cups is a dependency of the entire graphical subsystem, just removing cups also removes everything from the Nautilus file manager to Firefox to ubuntu-desktop itself.
The bugs found look correct and have security implications, but what is demonstrated is therefore not really "root privilege escalation" since it applies only to users who already have that privilege.
This post shows a way that clever code can execute anything it likes as root without knowing the user's password. That seems pretty significant to me.
If it has control of your user account, then it can just arrange to wrap your shell prompt and wait for you to sudo something else. The sudo password prompt in its default arrangement doesn't really provide much security there and isn't expected to.
On the other hand you probably don't need to. Those users already expose all the juicy data on the server. You don't gain much from obtaining root anyways, except better persistence.
This attack might be more interesting when chained with some other exploit that gains access to a users system via their e-mail client or browser. In other words nice if you're NSO Group making exploits for targeting individuals, but not that useful if you're trying to make a botnet.
Only in "advanced persistent thread" territory is your point really relevant, but the attack I describe is much more widely applicable. Having to wait a while is therefore not in any way a mitigation. In practice then, one cannot assume any security from sudo requiring a password.
https://en.wikipedia.org/wiki/Advanced_persistent_threat
(For what it's worth, I think most people would not lose much security from running as root, and the obsession with sudo is so much security theater, for exactly this sort of reason.)
And while lpadmin users can escalate, I’m more interested in escalations from services like web servers or whatever, running as low priv users. I use sudo to allow scripts running as my web server to run specific limited privileged programs as a simple layer of defence.
The first time you use certain flavors of sudo, you get a nice little message which reminds you why sudo exists:
Realistically, sudo exists to remind a user of these points. That is: by needing to type “sudo” before a command, you’re being reminded to pay closer attention that you’re not violating another user’s privacy or doing something that’s going to break your system.It should not be, but that's a different issue. It amazes me the amount of open-source projects that want to be installed with "sudo" when there is no reason they should not be able to be built and used entirely from within the developer's home directory.
I know more than one person who starts a shell session with "sudo -i" and then just works as root because typing "sudo" all the time is an annoyance.
For me, Linux was a hobby prior to and separately from programming. In the tutorials and documentation I read, every command was explained in detail and it was emphasized to me that I should never run a command that I don't fully understand. All instructions to elevate privileges were accompanied with advice about being careful running commands as root because root's privileges are especially dangerous. I was interested in those warnings, and took them seriously, because I wanted to master what I was learning. What I was learning, though, explicitly included Linux/Unix norms like security 'best practices'.
Developer documentation doesn't usually concern itself with Linux/Unix norms the way that tutorials for Linux hobbyists and novice sysadmins do. At the same time, the developers reading it might be perfectly dedicated to mastery, but just not really see what is considered proper usage by sysadmins (let alone the considerations that inform the boundaries of such propriety) to on-topic for what they're studying/exploring/playing with. Diving into those details might not be 'patt of the fun' for them.
What such a developer learns about sudo is mostly going to come from shallow pattern recognition: sudo is a button to slap when something doesn't work the first time, and maybe it has something to do with permissions.
But I think that comes from the mode of engagement, especially at the time of learning sudo, more than the mere frequency of use. I use sudo several times every day (including sometimes interactive sessions like you mention, with -i or -s), but I am careful to limit my usage to cases where it's really required. I'm not perfect about that; occasionally I run `sudo du` when `du` would suffice because I pulled something out of my shell history when I happened to be running it from / or whatever. But I certainly don't run it reflexively or thoughtlessly.
Use sudo instead of root. Change ssh port. Disable ssh passwords. Use some port knocking schemes or fail2ban. Close all ports with firewall. Some of those requirements might come from some stupid complicance rules, but usually they just come from rumors and lack of clear understanding of threat model. And sometimes they might even slightly reduce security gurantees (like changing ssh port to 10022, there's actually a security reason why only root can bind to <1024 port and by changing it to higher port you lose some tiny protection that unix greybeards invented for you).
I'm not saying that all those measures are useless, in fact I often change ssh port myself. But I'm doing that purely to reduce log spam, not because that's necessary for security. Configuring firewall might be necessary in rare cases when brain-dead software must listen on 0.0.0.0 but must not be available outside. But that's not something given and should be decided in case-by-case basis, rather than applied blindly.
They’re very different models of computing.