That's what I use for my personal stuff that I do. I like being able to cut and paste to reprioritise, etc. Also, when planning, having a real editor makes things go so much more smoothly. However, for people who are consumers rather than producers of the plan, I have found considerable friction about removing their drag and drop, table oriented UI.
I've often thought about writing a trello-like whose backend was simply markdown or org-mode files that was versioned in a real version control system. There is nothing in the Trello UI that really precludes it (and in fact, there was at one point a program that allowed you to talk to the trello API and grab/push org-mode oriented data from Emacs -- I never found it worked particularly well, though).
Org-mode and other text files have the problem that they can only present content in a linear way. Which is fine in a lot of cases and for text documents, but can't hold a candle to a simple Kanban board where you have everything at a glance and nicely structured without scrolling back and forth all the time.
I use org-mode files every day and it's a limitation I feel more and more.
Kanban relies on sensory intuition. Org-mode requires you to build up a whole new epistemological relationship to the elements of an outline. They are pretty different, and Org-mode requires a _lot_ of time investment.
I would go further and say that plain text (yes, I know org-mode is plain text too) would work just as well.
Each board is a directory. Each column in a board is a file. Each task is a line in the file. Open up the directory in the text editor of your choice. Move tasks with your normal cut & paste commands that you use for everything else. Keep the whole thing in version control.
We use images/diagrams/LaTex/etc to communicate a number of tasks that are inherently visual, not to mention needing somewhere on the cloud to access it centrally.
As a casual org-mode user, I fail to see how org-mode is really relevant for a team-centric tool.