At the Snow Sprint 2008 I started a project to build a light-weight page-centric web application on top of Zope 3, named Vudo. The effort was quite successful and our team managed to design and implement a very reasonable foundation. In this post I won't go into details on that particular project (instead see Christian Scholz' video interview). Instead I'll use it to argue that it's sensible to consider a rewrite of Plone, a content-centric web application, in the sense of a reinvention.
A decision was made recently to take an evolutionary approach to the continued development of Plone rather than a revolutionary one. I'll assume that at the very least a revolutionary approach would imply basing the application on Zope 3 which translates to a retirement of the greater part of the codebase. Quite correctly, the current path will eventually ease the move to a modern framework such as Zope 3, but I worry that it will come too late. It's simply too attractive to say farewell to the unruly, complex and often deeply troubling stack that Plone currently rests on. (I suppose I could find a thing or two about Plone itself that one could also quite easily depart with, but that's besides the point.)
It's true that Plone is already in its present shape a great solution to a lot of people besides a platform that has proven to be a viable source of income for many developers. It's worth mentioning however that both the solutions and the business in developing the solutions are challenged by the evolutionary approach. As it happens, the issues that we're facing with the stack are rooted so deep that we must continually write many lines of code to satisfy each customer and the solutions we build easily become moving targets for software deprecation.
I'd like to encourage the summit attendees to discuss the idea of developing what some people have already started to call Plone light, side-by-side with the 3.x-branch, representing the revolution if you will. The project should aim to depend on the already componentized Plone-packages yet still provide the playground for reinvention. At the same time the on-going debate on what should and should not be in the Plone core now has a chance to materialize.
14 comments:
I agree with you that the situation in Plone's stack needs some comprehensive attention.
At the same time, the course you prescribe sounds bleak. Zope 3 went down that route. I think Python3000 plans to go down that route.
Is it correct that the Plone-Light project would not support any exisitng Plone sites or Plone add-ons? And it would not support any existing Plone training or books? If so, and if the majority of core developer enthusiasm moved to Plone-Light, how would you ensure Plone didn't arrive at the unattractive position Zope 2/3 has arrived at?
I'd love to hear some ideas that mitigate it. I also would like a future for the stack that makes people enthusiastic. But I wouldn't be that interested if the response was what Martijn Faassen got when he questioned Python3k.
I too have had the urge to rewrite. In fact, I do it often (Poi, anyone?). I think it's a natural instinct of the developer, because we can see that it's possible, conceptually, we like the challenge and we like the promise of no-more-cruft.
Unfortunately, cruft is a result of the inability for anyone to design anything so complex perfectly the first time, and imperfect co-ordination between the large number of developers needed to actually get anything like this complete, and so would creep in again. Zope 3 was meant to be cruft-free. It's a great stack - I love it - but it's not cruft-free. Why do we have Grok? And why is Grok still being refactored? :-)
There are lots of things we could do, however I think the driving force needs to be at the other end: users, integrators and businesses who make their living on Plone. No matter how lovely it'd be to have a clean slate, leaving them behind would almost certainly kill off Plone. Mambo/Joomla split (for a different reason) and suffered, though Joomla lives because they had an order of magnitude or two more developers and users than Plone does.
Like Paul says, making all existing third-party products, books, tutorials, and - most importantly by far - knowledge in people's heads invalid is a recipe for disaster. Sometimes we need to stake out a path that makes sense for the coherence of the community, not just for the sensibilities of developers.
That said, I think there's lots of scope to revolutionise. I'm doing some prototyping of a new way of defining content types that should make the act of writing new content types for integrators and "light developers" a whole lot less painful. This is Zope 3 centric and very declarative, albeit with some CMF/Zope 2 integration cruft hidden away behind the scenes. It feels pretty nice to use though, at least in my demos.
Some of us will have to touch the deep, scary stuff and make it all work. Some of us need to be "integration artists" that make the tough calls and design the migration paths and write the documentation so that the rest of the world can just get on with it. To the rest of the world, though, that'll still feel like a revolution.
I haven't looked at Vudo yet, but it sounds cool (and I tend to like the things you design). I think the real future here is in re-use - of concepts and code libraries - and in data interchange. I hope you'll write more about the concepts behind Vudo so that the rest of us can learn, help them evolve and eventually see them integrated into a larger whole in which Plone - maybe not quite as we know it, but still Plone - is a major part.
Martin
Zope 3 certainly went down that route and although overly in danger at some point (or so I gather), today I think most of us appreciate the choice as being raison d'être in a crowded market.
The situation you're sketching out for Plone should the rewrite/reinvention be undertaken is very real. I don't believe existing add-ons can be moved over easily, nor will training/books apply unless they deal with the componentized part of the stack.
That may seem unfortunate, but if you look back over the last couple of years, a lot of the knowledge people are accruing is already deprecated. On the other hand, moving to Zope 3 will actually make it possible for a different crowd to apply their knowledge to Plone. And, to be bold, this knowledge is a better value because it's not a moving target.
I think part of the effort is to make development with whatever platform easier. Maybe invalidating some of the knowledge people have about Plone isn't so bad (think the x different ways of making forms in Plone).
Lately, Plone started to use quite a few of these reusable components. And reuse is good, everyone agrees; it also means better reuse of knowledge. There's just a lot of knowledge that's non-reusable and a dead end generally with Plone as it is today.
I posted a reply at http://vanrees.org/weblog.
(previous comment retracted because of mangled url...)
Reinout van Rees
That may seem unfortunate, but if you look back over the last couple of years, a lot of the knowledge people are accruing is already deprecated.
First, we haven't really told people it was deprecated. I think we've told people, "Sure, go ahead and use Archetypes, it is the correct entry point." Other things are experimental and still evolving.
That notwithstanding, you're proposing to go far beyond that "deprecate-but-still-execute" into "broken". At least I think so.
While I think an effort like that is worthwhile, and certainly more fun than the inherited band-aids, we would be criminal if we tried to call it Plone. I disagree that most people think Zope 3's break-everything approach was correct. I think most now feel that starting with Five, instead of ending with Five, would have been better.
(Actually, it would be fun to do a straw poll and get an answer on how people feel on that.)
Ultimately, it was misleading to call the result "Zope 3", as it ultimately gave up on being the next version of Zope, and wanted a clean slate.
If I was in charge, I'd do something like what Martin said:
- Follow Lennart's proposal for the layers of a new Plone
- Make a new top layer that was what the average people should work with. Try to make that layer support most of the Archetypes contracts.
- Once that layer becomes "Plone the framework", eliminate every layer underneath. Have no "Plone the framework."
Anyway, the point is: pick something that is the "right" top layer, stabilize it, and rebuild the guts underneath.
Any wholescale "start over" should not call itself Plone, IMO.
First, we haven't really told people it was deprecated. I think we've told people, "Sure, go ahead and use Archetypes, it is the correct entry point." Other things are experimental and still evolving.
I don't consider Archetypes to be integral to the problem. The objects are unfit and regular patrons of saturated fats but - which is more - they're inherently wrong. The good news is that we already have a model that is much more correct. Martin Aspeli is leading the way here and the development seems to take a healthy direction.
The real issue is that replacing the bad seeds with good ones while keeping compatibility is a huge undertaking and as a free open source software project it can prove too great a challenge.
As I understand him, Lennart wasn't proposing to add another layer but rather find the right audiences for each layer. It's too difficult to get a grip on all the layers of a web application and this is an argument for splitting them up roughly into audiences.
Perhaps a better approach would be to rethink the contracts that we want different objects to answer to including relations to other objects and then adapt the deprecated objects to these using a minimal layer of glue.
The effort that I call Plone light exactly implements these contracts and relationships, but as strict implementations and not as glue to the old codebase.
This is brilliant.
And don't get me wrong Martin, or Limi, or Joel, or anyone, it's not even brilliant because any of the things that Paul says are not true - they are true, except that the Zope2/3 situation is unattractive in comparison to not having Zope3, which most of Zope2 pretty much sits on top of these days.
The truth is that probably noone in the Plone community has more seriously considered a rewrite than I have. I often disagree with the social structure of Plone and though a long time contributor have rarely felt my ideas were welcome at the table when I wanted to bring them.
I've decided recently that most of this was just timing problems, I have an idea, race to the computer, find someone already doing it at least halfway as well as I want to, and I am really impatient about wanting to have influence over something, even if being patient is faster than starting my own thing from scratch. Oh, also, when an idea really needs a lot of championing and politicking, which goes without saying in case of a rewrite, I don't think there is much choice but just to go forward, from scratch. People are often more likely to give useful input as harsh criticism of how stupid your idea is than simply to decide it's worth their time to do it right the first time around.
I think that more of us should spend some of the time otherwise spent butting heads doing things entirely differently, I think a lot of the unspoken heroes in the land of Plone have innovated this way, just by leaving brilliant ideas in their wake that Plone is awesome enough to be able to adapt to.
Is AT the problem? Is Zope2 the problem? Maybe it's cPython and the GIL, I have chased that for long enough. Maybe it's the whole darned thing, right? What's worth saving in that case, though? Is compatibility important? If so, with what? With all AT? AT since version n? Howabout all CMF apps? A lot of people use non-Plone-specific apps with Plone and love it.
For Plone to evolve, it will have to spawn some failed attempts at betterment, I think the problem with the current evolution slant is that we are picking up a lot of cruft that looks more like an appendix than an opposable thumb. Because Plone is software, we can just delete the appendix, see what unit tests fail, and come up with several approaches in 1/100 the time that it takes the human race to even attempt one miniscule generation of change and observe the result. ;)
What is really interesting to me is that you have a completely rethought idea, and a very good one, for the top layers of Plone. I have come 'round to wanting to see Plone run on Mono/.NET, and in fact not even primarily using IronPython or any such. Zope3 might just be clean enough to be patched into compliance for RPython without too much trouble, allowing to make a 'zope' package available just as if it were written in C#. It can only called into the core library of .NET, but anything from C# to IronPython can call it, and the underlying platform implements a lot of stuff that Zope3 painstakingly provides for us, like events.
Plone could probably run, in this case, without a ton of modification using IronPython or PyPy, and I have a nutty theory about implementing Zope2 legacy APIs, which mostly just call into Zope3 by now, using C#, which would significantly reduce the overhead of legacy code, and possibly even make room for restoring some deprecated apis and allowing more versions of Zope code to run together at once.
I contacted Limi a bit late to make the summit, and probably should not take up a spot anyway, as I have to be in class for part of every week day, but I am glad to hear that someone will be talking about revolution.
Another project I have aims to implement some sort of behaviour that Plone has grown away from, in a Plone-friendly way, and I was going to ask folks whenever an opportunity arises, to collect a list of things plone doesn't want to do, or even hates to do. :)
Cheers!
Justin, trying to avoid getting distracted by the the obvious arrogance of your post (you do know you don't have Plone core commit access, right?) - if you want to go off and write a new CMS from scratch, by all means, go ahead. You can't call it Plone, though - Plone is trademarked and protected by the Plone Foundation and owned by the Plone community. You can't call it "Plone Light" either. Or "Plone NG".
Why? Because while you're having fun trying to get Zope to build in IronPython, the rest of the world is trying to take Plone seriously. If you try to get a large client to consider Plone up against established commercial competitors, this kind of talk is scary as hell to them. If I invest in Plone today, will the thing I invested in be unsupported and dead in a year? If I train my developers in this platform, will their knowledge be obsolete and non-transferrable soon? Believe me, anything without a full migration path is DOA in terms of existing customers and probably DOA for most new customers too. It's not like we don't have any competition in the CMS space. :-)
Secondly, to inject a dose of realism: if you started this project today, even without any of the IronPython/PyPy/whatever-is-hot-today earthquake shifts, I would be incredibly surprised if there was anything actually useful, stable and dependable (and that means - with some degree of popular support and documentation) in anything less than two years.
Plone can't afford to stand still for that long, and we don't have the resources to support two completely separate projects and mop up the confusion and co-ordination difficulties this would cause. Look at Zope 3. Zope 2 development basically stopped for a long, long time, and only picked up again because we were trying to migrate to Zope 3 and Five.
As I said, I'd rather we started in the other end:
- define how we think people's experience of Plone should be, at different levels of expertise/audiences
- make clean APIs for that
- design sensible, realistic migration paths to that, where appropriate
- glue together
Of course, efforts like Vudo are part of the work that goes into figuring out how this needs to work in the first place.
Once we have that working, we can refactor the internals and deprecate the stuff we don't want people to use anymore. Maybe if those APIs are rock-solid, we could rewrite the insides from scratch. But only when they are solid and understood can we do that.
By all means - experimentation is hugely important, and as Godefroid said in Naples - we need to fail more. However, there's a fine line between throwing everything (or just "too much") at a reckless project that will, I guarantee you, be harder technically, socially and politically than it seems when you're breathing thin air and drinking schnapps. :p
Oh, and if I'm not mistaken, someone started rewriting Plone at the last snow sprint too. Does anyone remember Cubed?
To Malthe - I feel Justin has probably hijacked your idea here a little and stretched it too far. I would still *love* to see some more technical discussion about Vudo and how it's different from Plone in terms of design principles.
Martin
Even taking account of all the new components, I think a Plone-Light would be incredibly difficult to achieve – there are just so many man years of work in the current integration.
It seems clear to me that the future of building content types with Plone lies with zope.schema. Support for this method of development is progressing rapidly, but it's not there yet. It's important we get the content type developer story for this right, but when we do we should move the core content types over - probably with a dump/restore migration.
We can't abandon Archetypes quite yet. Both as developers individually and as a community we have too much invested in it. There must be a significant period of time (years?) where it continues to work and integrators can mix and match, picking what's right for their site.
Evolution not revolution. Not as exciting maybe, but more enduring.
With blobs and a scalable catalogue implementation looking within reach, the future of Plone is bright.
viva la revolution..
the gradual integration of zope3 into plone has made plone more complex to develop and debug, and has lost imo the promise of zope3, clean components, reuse, and simplicity. we've gotten integrated it at the cost of complexity, which is self defeating.. imo
we're increasing the learning curve, for new developers.. by introducing additional technologies and complex interactions, instead of simplifying the core.
we can make plone product all slick and shiny, but if you want to know why drupal, rails, django have had a great success.. its because at a core they keep it simple for developers to get things done.
i think zope3 dev cycle is held up a strong man argument too much.. the fact is that impressive systems can already be built on it, if people are willing.. launchpad, hivurt, lovely are all evidence to the same.
the fact is we won't have a useable zope3 cms, till we build it, and stop dragging around the dead bodies of zope2 and archetypes. so let's stop wasting stop energy on folks trying to make a useable platform.
Kapil - I think you're right that complexity has increased, and that some refactoring is in order. However, complexity has increased along with capability and power, so it's not exactly lost effort.
I have no intention of applying "stop energy" to anyone wishing to write new and unencumbered code. However, we need to assert that this new thing can't be called Plone (or "Plone Something"), at least not until after the writing, integration and migration path are ready and available - for all the reasons Paul, myself and others have stated here and elsewhere. That doesn't mean we can't share communities, though. ;-)
Personally, my energies will go into what I ultimately believe to be an endeavour with more potential for success: re-thinking the Plone third-party development model in light of modern frameworks and concept and attempting to support it in a layer that offers some isolation from underlying cruft whilst still affording all the power that Plone gives you out of the box today.
FWIW, I agree with Kapil about the complexity. I often tell people that I wouldn't wish Plone development on anyone who wasn't familiar with Zope2 before the injection of Zope3 technologies, because it just doesn't make any sense to regular "humans". There is too much "backstory", and a lot of conflicting technologies playing ball in the same field (Acquisition plus the Component Architecture makes my head hurt).
Having said that, I can't think of a better time to be a Plone® consultant than right now. Like Martin says, Plone is about the end user, and any developer that loves Plone-the-software will learn to love Plone-the-backstory.
Well, maybe not, but (hopefully) they will be interested enough to take it to the next level, because they believe it is worth the learning curve (and they would be right).
If I were learning the stack today I'd be curious about the same kinds of things:
- What is Zope? What can I do with it by itself?
- What is the CMF? What I can I do with it?
Today's stack inspires the same type of curiosity (and then some). We just need to continue to evolve (revolve, whatever) by making the best possible decisions with regard to delivering the best possible Plone out-of-the-box experience we can, however we can (ugly stack, rewrite, divine intervention).
My 2 cents (FTW),
Alex
Post a Comment