> As an observer it looks like Python's package ecosystem in particular is a giant mess.
Most people would say it is. The better question is how to avoid such a mess.
My take is that maintaining backward compatibility is a core principle which needs to be strictly observed to solve that problem. And yeah, that has also become a cultural issue with Python, as the Python2/3 breakage shows.
So, could one sum it up in that Nix magnifies unsolved compatibility issues in packaging systems? Because if there were a single core Python distribution, like say, Anaconda, but nothing else, these issues would not exist. Of course, people can avoid the issues if they only use a handful of packages. But putting all into a single channel makes the problem much more acute.
> My take is that maintaining backward compatibility is a core principle which needs to be strictly observed to solve that problem.
Given that software developers never guess the correct design up front this means that you always have architecturally buggy software, and a bunch of complaining about why buggy-looking edge conditions are never fixed. There has to be some kind of release valve for software to evolve and break backwards compatibility.
I don't disagree, but it doesn't only have to go in this one direction. One of the most interesting things about Rust for example is how it tackles experimental implementations and has concepts in the compiler etc. that make unstable language features "first class". I'd say this will definitely yield better results than "well a couple of guys hacked around on some prototype forks of a compiler, and now we're stuck with the result".
Of course, they also make very impressive backwards compatibility guarantees for stable stuff (cf. Rust's "editions").
Rust has corporate sponsorship and a very experienced team of developers.
You won't get that level of attention to detail and commitment to getting the design right up front, and the willingness to maintain old APIs in the name of backwards compatibility in a single-person open source project published into a package manager done on someone's free time.
So you are probably arguing for very thick standard libraries which are maintained by the core language team, which is corporate sponsored, and a reduction in reliance on open source packages.
That also means as well that we shouldn't tolerate "shaming" of projects for taking a long time to fix and merge features since 95% of the work will be required to be done up front in thinking about the right shape of APIs.
I'm cool with all of that as long as the whole package comes along. The idea that a bunch of solo, unpaid open source maintainers are going to be doing good API design up front and maintaining perfect backcompat, while being incredibly responsive to PRs from the community is kind of "unicorn farts" levels of not going to happen in the real world. You sort of get what you pay for, and a bunch of unpaid solo volunteers are going to need to make breaking changes to fix their old mistakes and abandon maintaining their old tech debt. And if you paid nothing for it, really you're getting more than you deserve in that deal.
> So you are probably arguing for very thick standard libraries which are maintained by the core language team, […]
No, I'm not arguing that Nix should do anything in particular.
All I was saying is that the "we have to get it right the first time without much feedback" way obviously isn't the only one and there's empirical evidence of other working models.
As for PRs and so on you're really putting words in my mouth, and frankly, I don't like it. Just so you know. I have never made PRs to core Nix, but for Nixpkgs I have only had a good experience so far.
---
Edit: Or are we really talking Python? In that case, I could even less comment on PRs. But: Python has a large stdlib (it's "batteries included", after all). But also, Python often has found good ways to deal with its warts.
And I hope I don't have to argue that Python3k wasn't worth the trouble, right?
And frankly, there, I'd argue that growing to the point that python has you'll have to reexamine some more ways to gather data about community interest, for example.
From the outside, the process around the walrus operator and Guido leaving the BDFL post looks like a prime issue of either not having enough "wild information" early on or of the final decision ignoring a vocal part of a huge language community.
> There has to be some kind of release valve for software to evolve and break backwards compatibility.
You kinda insinuate that breaking backwards compatibility is kinda necessary at times.
This is not the case. Projects like
* the Linux kernel, or
* the GNU C library, or
* the Numeric -> Numpy transition around Python 2.0, or
* Common Lisp (which is much older than Python) adopting Unicode
are good examples that this is not necessary. It is not true that you have to break backward compatibility.
There are domains where breaking backwards compatibility in libraries is not acceptable at all, like vendor libraries in industrial automation. You don't throw away a 15-year old printing machine or a chemical plant just because the vendor of the automation software is tired of supporting its old interfaces.
How it is done? It starts with well-designed interfaces. And when interfaces are changed, the old interfaces are kept and become special cases of the new ones. Numeric/Numpy is a good example.
Here is a talk, brilliant as always, by Rich Hickey which explains why and how:
Python3 could have gone the same way - keeping the interpreter compatible to Python2 code, making the semantics dependent on whether a source file has a *.py or a *.py3 extension, and so on. It would have been more work but the transition would have been nearly painless, and I guess much faster. Support for old stuff does not need to go on forever - for example, Linux does not support any more Intel 386 CPUs.
It boils down to whether keeping stuff backwards-compatible is a goal of the project leaders or not.
The problem with most open source software that is in package managers is that it is usually done by one person. It isn't started by someone with a decade of interface design, it is often their first large important project, they DONT do the well designed interfaces because they haven't made the mistakes in interface design yet which they'll eventually learn from. And then when it comes to backwards compatibility it is cheap for you to say they should just support their old interfaces forever, but that has a cost and creates more friction going forwards for the projected. When it is one person working on open source who isn't getting paid, that is all somewhat unreasonable to expect and you just won't ever get it. In your world what you'll wind up with instead of back compat breaking changes is just abandoned and rotting software as maintainers give up.
You could do this by arguing that languages need very thick and well-designed standard libraries, which means that hopefully there's a large business supporting the library and there are teams of reasonably well paid software engineers who are doing the design work up front for everything. You should be explicit about that though.
I'm kind of not surprised that you cite one of Linus' asshole rants to LKML as well. Try screaming that at a single-person open source maintainer and watch them decide it just isn't worth it any more and quit on the project entirely.
If you want that then don't use anything outside of your language's standard library and don't use package managers and contributed source code at all. Write everything else yourself, no dependencies, no worries about backwards compatibility breaks.
I did not say that hobbyist packages which are used by few people and are unstable experiments should be kept stable at all costs.
But you see the linked discussion about stability in Nix is about packages like opencv, pillow, boost, pytorch, tensorflow, kubernetes, and I would expect them to behave professional.
And as said, as long too few people actually respect semver, it is pointless to suggest to use it, especially if the authors of a package do not know what a breaking change is, do not know how to avoid them to happen, and do not have a documented and specified API in some way. If you don't have an API, you can't use semver.
I do not complain. Complaining is like "Uncle Bert was totally drunk again and fell down the stairs and broke is arm and I expect him to change in order to make me happy." Or "Frank did lend my car again and damaged it, and I told him again that I do not like that".
What I do is observing things and drawing consequences. "Sorry, Frank, you can't have my car." And: "Well, Uncle Python does of lot of breaking changes, so I do not better use it for long-lived projects which I do not want to constantly fix. Maybe I could have a look around what languages do manage this better?"
This is not, I think, an attitude I am alone with. For example, the Python2 / Python 3 breakage led Konrad Hinsen, an earlier contributer to Numpy and Scientific Python, to explore Racket as a language for scientific computing:
And more concretely, and perhaps pragmatical, when I start a project or include a library, I am critical about the stability the environment offers. For example, in some situation I used gevent in place of newer and perhaps fancier solutions because it was stable across the Python2/Python3 version bump, and I did not want neither myself or my coworkers to need to re-write that part.
This does not mean I stop to use Python altogether. It is still useful for many applications. However for writing new library code, I will rather use something that is more likely to be stable.
Well they should clarify because they cite the example of the 2/3 breakage which is a major version break, and they're not complaining about packages just violating SemVer so I don't think your interpretation makes sense.
It also certainly wasn't what I was responding to, and responding with "hurr durr major versions" like I've never heard of them before is just mildly insulting (and kind of insulting to the parent comment by proxy)
> they're not complaining about packages just violating SemVer so I don't think your interpretation makes sense.
Packages regularly violate semver, to the degree it becomes a cargo cult.
It is funny that at the one hand side, people say that keeping backward compatibility is too difficult for normal package authors and contributors, and on the other hand side suggesting that using semver would improve this.
In order to actually use semver, one needs to know what backward compatibility is, what kind of changes break it, and how to make sure that this breakage does not happen. It is not that difficult. But also, semver require stability against an API, so one absolutely needs to have a clearly documented API of some sort, because otherwise, if there is absolutely nothing specific you promise, how could one expect you to keep it?
And further more, major packages should actually respect semver if they claim to use it, and not do breaking changes with minor version numbers, like for example boost does. Actually, I think if somebody uses a three-element version number and does not strictly adhere to semver, it should come in a popup box in front of every download link, because these three-part version numbers somehow imply that the package uses semver, and this in some cases (like boost) is a false promise.
And before somebody throws in that it is not *his* package that is breaking semver, but some dependency that his package happens to use: No. If you use dependencies, you are responsible for their behavior, because otherwise, one could always shift the blame somewhere else. If a dependent package breaks backward compatibility and your package is a library package, including it is a breaking change, because backwards (in)compatibility of dependencies which have visible effects (as is the case in all Python library modules, as has been discussed) is a transitive property which travels up the dependency graph. If you include visible breaking changes, then your package introduces a breaking change, and cannot honour semver without bumping the major version number.
> they're saying that the Python 2/3 breakage debacle became a model for how packages are maintained in general.
As explained above, semver is not a solution. And it is often not really followed. For example, boost is breaking backwards compatibility some times, and this is causing problems, last not least because boost Python bindings are used in so many projects.
explains why this is not a solution. It makes a difference, yes. But it is the difference between "the incompatible changes in my library are going to break your application" and "the incompatible changes in my library are going to break your application, and I am telling you this beforehand".
Is the discussion here that Nix doesn’t support multiple side by side installations of different versions making it difficult to install two packages depending on the same package but different versions ?
I thought in Nix you can have that, and python can have venv for independent library versions, is it that no one has done the work to combine the two ?
Nope. All Python programs in Nix are effectively venv'd, so-to-speak.
The problem is that if two packages A & B each depend on the same library L but they require different versions of it, any Python code that imports both as libraries will end up with two, potentially incompatible, copies of the same library in its import path. And Python doesn't support this, so it uses the same version of L with all the Python code running in that process. So now depending on whether some package P which requires both A and B imports A first or B first, A will end up using B's version of L or vice-versa. This sometimes does nothing, and it sometimes causes very weird, very subtle breakage.
Thus for all of the Python libraries in Nixpkgs to be usable in any combination by any package in Nixpkgs, there can only be one version of each Python library in Nixpkgs.
Once your package collection is large enough, you start actually encountering versioning conflicts as described above in the transitive Python dependencies of your end-user application packages. That's why Linux distros run into these integration issues. Application developers generally don't because their applications development environments are much smaller.
> I thought in Nix you can have that, and python can have venv for independent library versions, is it that no one has done the work to combine the two ?
A python environment is effectively a different take on a venv. Python packages is one giant python environment. For programs that life outside of python packages but use the packages from there we are free to apply overrides how we want. So it is possible to have different versions of a package in a different python environment.
Most people would say it is. The better question is how to avoid such a mess.
My take is that maintaining backward compatibility is a core principle which needs to be strictly observed to solve that problem. And yeah, that has also become a cultural issue with Python, as the Python2/3 breakage shows.
So, could one sum it up in that Nix magnifies unsolved compatibility issues in packaging systems? Because if there were a single core Python distribution, like say, Anaconda, but nothing else, these issues would not exist. Of course, people can avoid the issues if they only use a handful of packages. But putting all into a single channel makes the problem much more acute.