Question, if I have a CSV file that I'd like to do some quick SQL queries on before moving the results into Pandas. What would be good resource to do this?
Preferably compatible with the rest of the Python-dataframe ecosystem and as simple as pd.read_csv()
SQLite is what you're looking for. If you want to use the CLI,
.format csv
.import <path to csv file> <table name>
Alternatively, read your data into pandas and there's extremely easy interop between a DBAPI connection from the python standard lib Sqlite3 module and Pandas (to_sql, read_sql_query, etc.).
I would second this suggestion. I was querying CSV data imported into SQLite this weekend, and it was extremely easy to get started. SQLite is pretty snappy even for tables with millions of rows.
SQLite supports defining columns without a type and will use TEXT by default, so you can take the first line of your CSV that lists the document's dimensions, put those in the brackets of a CREATE TABLE statement, and then run the .import described above (so just CREATE TABLE foo(x,y,z); if x,y,z are your column names).
After importing the data don't forget to create indexes for the queries you'll be using most often, and you're good to go.
Another suggestion for once your data is imported, have SQLite report it in table format:
If you want to try it in SQLite (pros: no need to run a server or install anything since it's in the Python standard library, cons: not nearly as many advanced statistical analysis features as PostgreSQL) my sqlite-utils CLI tool may help here: it can import from CSV/TSV/JSON into a SQLite database: https://sqlite-utils.datasette.io/en/stable/cli.html#inserti...
The sqlite3 connection object in Python allows you to register callables which you can use as scalar or aggregate functions in your SQL queries. With this, you can fill some of the gaps compared to PostgreSQL by essentially importing Python libraries. I just found this nice tutorial while looking for relevant docs:
However, I think the limited type system in SQLite means you would still want to extract more data to process in Python, whether via pandas, numpy, or scipy stats functions. Rather introducing new composite types, I think you might be stuck with just JSON strings and frequent deserialization/reserialization if you wanted to build up structured results and process them via layers of user-defined functions.
In addition to the recommendation for sqlite, I've found `csvkit` to be an extremely useful set of CLI tools for CSV munging. The `csvsql` [1] entrypoint is especially handy because it allows you to issue SQL queries against your CSV file directly vs. loading then querying.
I keep a non-commercial installation of SQLServer on my machine for this reason, but this is likely overkill for most purposes and requires a Windows machine.
It does have some nice import features for CSV data though.
Congratulations guys. I met two of you when you visited our agency and told your story. Very impressive, and for those of you reading, these guys were still in high-school when we met them, maybe still are?
Stoked that you got into YC!!
Author here: Sheetfu original goal was to port the Google Apps Script API for spreadsheet (made in javascript) to Python, so you'll see a lot of common ground between those 2 libs.
You could probably achieve the same with both Pygsheet and Sheetfu, the interface is a bit different. Sheetfu, however, contains the Table module, which completely abstracts the coordinate system for a more ORM-ish syntax.
- The guy presenting this might be wrong.
- He may be an asshole and/or have resentful intentions with making this presentations.
However, none of the people who disagree with him in this article engage with any of these things. They're just angry that he's saying this at all! Their upset emotions seems to be the only case against what he's saying.
- Private corporate memos written after feedback was explicitly requested
Probably others.
The same thing was said in every case: "it's not the right forum, it's insensitive and without empathy".
How incredibly surprising that there doesn't seem to be any right forum or right time for expressing opinions about anti-male bias. Somehow it's always offensive and it's always terrible that women were upset.
It looks almost as if some people want to shut that conversation down wherever it happens.
It seems as if that conversation was part of the bias in the first place, so there's two sides of the same medal.
One right place would have been the room where the anti-male/pro-female responsibility was decided. Now that the decision is through, it's just bickering and not constructive to oppose it, ignoring it denigrating it in public as if that was outside the assumed responsibility for the forum (the institution).
I can't speak for the libertarians and how they respond. I think a libertarian blog is the perfect place for it, and I'd have happily engaged in a debate, just like I am here.
I was very outspoken in the wake of James Damore being fired from Google, because I think there was a reasonable narrative in there somewhere, even though some of the science was flawed. I still don't think firing the guy was the right response. To my mind his ideas were respectfully presented and weren't phrased in absolutes. There could have been a great conversation around gender in Google as a result.
But this is different to that. This is somebody writing off a large group's experience, supported by poor science, in an area he knows nothing about, at a conference for the express purpose of supporting women. I don't want to shut down conversation, but I sure want to condemn people who can't be respectful of others.
Upsetting anyone? Probably not possible, and probably not desirable. There's always going to be people getting offended when you state a controversial opinion. But of course you can minimize the offense, and it doesn't take much imagination to think up a better option than a conference designed to support women. Blogs/journals/conferences not specifically for supporting women/editorials/radio shows/record a podcast. Write a manifesto for crying out loud.
His findings? A request for comment to the "Frauenbeauftragte" (gender equality officer, women's affairs representative, whathaveyou) or a prof of gender studies would have given a chance to compare his results to others. It's not exactly a master thesis or anything.
I think the selection of studies was somewhat myopic, and I thought his reasoning was flawed, so I'd change that.
But honestly I wouldn't have so much of a problem if he presented the same thing anywhere other than a conference for encouraging women into science. If I were any one of those women, I would have walked away discouraged and hurt.
Galileo maybe isn't the best example, since his reputation seems to (as I understand it) be massively overblown in an enduring myth.
But I think the question you're asking is: am I critical of him because it's fashionable today to push feminism with blinkers on? And would I still do so in 50 years, if today he sparked a mens' rights revolution?
Let's not assign this guy mythical status preemptively. There have always been loud points of view; some of them haven't held up with time, others have. We usually forget the ones that haven't, unless they managed to really hurt people along the way. To put it another way, just because you take a loud contrary position on a political or social issue, doesn't mean you're right.
I actually do applaud him for standing up for something, but my admiration only goes that far. You can present your point of view while trying to minimize the collateral damage. You can also be aware of your own fallibility.
If Galileo burst into Catholic prayer to proclaim the church was wrong and he was right, yeah, I'd call him a bit of a dick. What does he hope to achieve, by choosing the forum that is most disrespectful of others?
Considering the prevalence of the church and its properties at the time, there really was no “correct” forum for Galileo to promote his findings - everywhere is disrespectful! He was pushed out of every forum he went to.
This guy has the same problem - you will always just tell him he’s in the wrong place.
Well no. You're assuming a lot about me. I've expressed an opinion on exactly one place which I think is wrong.
You're probably right Galileo didn't have a platform, but you're the one drawing that comparison, not me. Do you really think he has the same problem, in the age of the Internet? In another comment below I've suggested other, more respectful, platforms: "Blogs/journals/conferences not specifically for supporting women/editorials/radio shows/record a podcast. Write a manifesto for crying out loud."
It's a mental illness, and while the article focuses on schizophrenia (and depression), they generalize it to all mental illnesses.
I'd be a bit less angry if they hadn't generalized it like that. A bit: how many "healers" have been lost to their visions, or to their depression? I have a feeling they'd simply sweep that under the rug as a natural consequence of seeing the spirits that others can not.
The best way to look at it is a corrective to the harmful pathologization of unusual mental states by psychiatry. Yes, the new-agey claims are not really plausible. But mental illness is less prevalent in traditional societies where it is understood in a more nuanced way.
AFAIK dementia patients don't generally get involved in the oppressive and demeaning psychiatric system. It's a horrible business.
It is important to listen to the experiences of the "neurodiverse" to understand how we need to make life better for them. It's true that schizophrenia is generally a disaster that destroys the life of the sufferer and the people around them. What needs to be more widely recognized is the psychosocial aspect of illnesses like schizophrenia, e.g. the extent to which they are caused by toxic families and/or social circumstances. This is where "healing" becomes relevant: a psychotic problem can be an opportunity to change and escape from a toxic situation. Of course, it often has the opposite effect, when the patient is pathologized, medicated, and told that they are the problem.
Those "unusual mental states" are not classified as problematic by psychiatry until they adversely affect the patient's ability to live in modern society. And if they can't deal with life in a modern society without assistance, something needs to occur. If the person has the support of their community to make up for their shortcomings, then great. If they don't have such a community, then if medicine, CBT, or can help them, why not use it?
As for how dementia is handled in the US? With full time care in lockup facilities. The few families who try and handle the care themselves fail most of the time, since it requires around the clock alertness and care, lest the person with dementia hurt themselves at 3 in the morning.
Also, most of the "neurodiversity" and "unusual mental states" proponents try to tell me that my ADD is a benefit to my life, that I would have made a great hunter gatherer. Which is, as the person actually suffering from the ADD on a daily basis, complete and utter horseshit. I know I couldn't have survived in such a lifestyle, because I'd die of starvation if I was required to hunt to survive.
Hunting requires moving in silence; ADD means I'm constantly bumping into things. Hunting requires sometimes interminable amounts of waiting; I'd get antsy after ten minutes and have to move. Hunting means being able to concentrate so when the right moment comes, you're ready to strike; ADD means I'd be thinking about ways to improve my spear and bow as a buck passes me by.
As someone who lived in Kochi for six months in 2004 when I was 19, I find this really interesting.
Do you live there now?
I remember we were going to leave Kochi and go for a trip to Munnar (the hills where all the spices grow and tea estates are), but on that day the labor unions had declared 'hartal', which was somekind of statewide halt which meant that if you were driving on the road that day, it was open season for people to throw rocks at your car.
Our guide just wiggled his head, smiled and said that we'd be fine :) And we were. Saw no signs of rock-thowers by the road. But instead our chauffeur drove like a maniac up those hills in that old Ambassador, and I didn't have any medical insurance at that time. That's when I learned the art of praying :D
I don't live there but I do have family there and try to visit whenever I can.
Hartals, an effective non-violent form of protest against British occupation, is a double edged sword. Now its mainly used to bring the country or state to its knees. If there is any violence it is usually politically motivated or by militant student movements. It is a weapon used by both the left and right sides of the political spectrum. But generally non-violent.
Bad driving unfortunately is universal. I wouldn't risk driving there myself.
What's interesting is that Kerala was the first state to ban Bandhs (a decision upheld by the Supreme Court). A similar measure to enact a total ban on Hartals failed.
Thought this comment on the NPR article was really insightful:
There's so much more to developing a healthy microbiome than just eating yogurt and drinking kombucha, as many people like to claim. It starts at birth. As you travel through the birth canal, you come into contact with many different and important microorganisms through contact with your mother's vaginal fluids and even fecal matter. If you're born via C section, you're starting out at a disadvantage. Then breastfeeding is the next step. Mother's milk is where you get your next dose of a very broad spectrum of microorganisms. Then throughout life, as you come in contact with other people, you eat dirt, you hang around farm animals, etc, these are all mother nature's way of helping you develop that microbiome, which is essentially the core of your immune system. Eating is important too, and one of the biggest problems with our modern food system is that everything is pasteurized, cooked all the way through, sterilized, irradiated, etc. All of the healthy microorganisms that used to exist are being killed.
So the point here is that people need to rethink many of their lifestyle choices if they want to heal their guts. Every time you take an antibiotic, you likely suffer much collateral damage because you may be permanently killing beneficial species that you may never be exposed to again. They don't exist in yogurt or kombucha. Either that, or consider fecal bacteriotherapy (fecal transplants) since it seems to be one of the the most promising new approaches to healing microbiomes. ~Julie Latham
Totally non-scientific response to this, but I can say that after having to take antibiotics a few years back for a pretty bad sinus infection my digestive system has never completely recovered. I never had any sort of digestive issues that I can think of but have had numerous since then. I can't directly contribute it to the antibiotics, but it certainly seems to correlate.
Taking probiotics helped quite a bit, which I did a month or so after the antibiotics started causing these issues, but yeah, never completely felt 'better' since then. I guess this is yet another example of why/how over prescription of antibiotics is a bad thing.
>as you come in contact with other people, you eat dirt, you hang around farm animals, etc, these are all mother nature's way of helping you develop that microbiome
I find that way of personifying our environment counter productive. It's not mother nature's way of helping us, it's that we evolved over millions of years in this environment and to change that is to hinder ourselves. I find this a much more accurate and convincing way of arguing for a back to nature style of development and nutrition.
