I saw the title and instantly thought of the "Load bearing mac mini" at Twitter.
In our server closet there was a mac mini sitting on another rack mounted server and plugged directly into a switch. IT found it, asked around and nobody knew what it was, so they unpugged it. Immediately the whole of engineering and support were basically offline.
Despite the thing looking suspicious as possible, I had set this thing up as an employee a year before. We were not allowed direct network access to our hosted prod network so as a "stop gap" I setup a SSH tunnel that listened on the mini's IP. At first we used this for access to the support web interface so it could be taken off the internet. At the time my request for a server was rejected. One by one more things got added to the list of things proxied over the device, eventually including basically all internal pages, git access, and about a dozen other random services. I finally got it moved into the server room, but not to real hardware. Once we built a DC we got peered access and the mini finally died.
It's an effective way, but generally a very bad one. I used to work in telecoms, and some idiot took down a tunnel between the UK and Germany carrying 1kB/s of RADIUS traffic because someone (the idiot) owned the link it ran on and had failed to record the formal allocation of this tunnel to a production customer-facing public access WiFi service I ran. I on the other had remembered to record his telephone number and his boss's telephone number, and a certain amount of screaming went on. Moral: before doing a scream test, make sure your own documentation is watertight.
I once took over a decommissioning project of a DMZ set up for a connection to a third party no longer used. Everything documented showed that all connections were no longer active.
Step 1 was to do the ‘scream test’. Some how the enterprise routed the most random traffic through there as we took offline random servers and people’s desktops. Turns out the screaming that happens is at you for making things break instead of a pat on the back for discovering bad networking.
One of the last contracts I worked, they provided such awful hardware that the company and employees ended up BYOD. Not enough memory for a modern IDE to start up levels of awful.
As the project ramped down they ended up having to buy an iMac off of us. Someone stopped using it and before the case even got cold I had installed a CI build agent on it to run Karma tests. Which ended up being invaluable but also never managed to get replaced with a real piece of hardware. So they just dragged that thing around as the project went into maintenance.
I always thought the load bearing Mac mini was a myth, it's such an impossible story and it's been so widely shared. I guess it's just a case of programmers taking the easiest route and having it backfire.
"Shadow IT" is the official name for circumstances that lead to this and it's the consequence of:
A) Deprioritising "non-urgent" but "important" tasks consistently
and
B) IT being bureaucratic and/or unable to allocate budget
I had a Mac Pro (trashcan) at Ubisoft that was the only way people were able to play our studios live game from within the office. (Ubisoft had a "NO OUTBOUND CONNECTIONS TO INTERNET" policy for Studios)
That same Mac Pro was running our internal slack bot to run Maintenances (and, insult people).
I left Ubisoft 5 years ago, as far as I know that Mac Pro is still plugged in on my former desk chugging along, last time I asked my former manager about it was last year.
----
At another company (now owned by Oracle), we had an internal IRC (this was before Slack) and nobody thought about it.
One day it went down, and traceroute had indicated it was in our server room; after checking every single server we could not find it, until someone noticed an ethernet cable that went through a run into the floor but didn't come back out.
After opening the suspended floor we noticed a laptop running Solaris. That was our IRCd and the OS had an uptime of close to 8 years.
Yet they forget to put a password to BIOS so I am able to write this comment from a school computer, on NixOS running from a USB drive. Though I'd probably crack it even if they did.
Don’t forget IT ignorance. A lot of places IT is full of Microsoft MVPs that have no understanding of the needs of people developing in and for Linux systems.
I would have thought so too, before I worked on a team where we had our own load-bearing Mac Mini installed at a client site - in that case, load-bearing for A/V reasons (no one was able to get PulseAudio working in time).
... At the end of the the deal with had two mac mini's with auto fail over configured via health checking. Thank goodness we got rid of the service provider that refused to let us pair with the network. Once that happened the networking team could just do normal peering with a standard router.
But yea, in the early days that mac sat on my desk. It only got moved when I pointed out the issue to our new security team and their jaw hit the floor. =)
Most startups I deal with these days don't even have offices anymore, let alone network ports. =)
But yea.. same deal with wifi. Its amazing how often the wifi password is posted on something visible. In fact I have found the password in so many public images which means somebody on the street could just connect to the wifi network fomr the street.
We have a labelmaker in our hackerspace. The box containing it has a label on it "Please do not print unnecessary labels". Below it is a label reading "ok". :)
Hey friends, use 802.1X for your datacenters and enterprise (certificate) wifi auth so you can audit and rotate credentials to prevent... exactly this problem. Everything else iot-like gets a very limited VLAN / alternate WiFi network.
Good points. The problem is, there are over 1000 people coming and going every day, the site has a BYOD strategy and the IT team is 4 people. We tried implementing 802.1X for LAN devices but it was soo much overhead that we dropped that.
The thing of this case is that the person was only able to place the Pi there because he had a key to the network closet. That's game over no matter how many security protocols you implement
OP here. What I didn't mention in the article is that this actually happened in a public School (small-ish for US standards as there are just ~1000 students and 100 teachers)
Hard to get the budget for serious switching hardware, even harder to get people who know how to manage them as I'm just an external contractor but can't exceed the alotted budget for my work there
Is WiGLE a thing people know about? How does that work, its a service that listens to all publicly available wifis in the country? I understand how it could be possible in theory but how is it actually practical?
Data is crowdsourced and contributed by users. On a much larger scale, Apple and Google collect data from iOS and Android devices to power their WiFi/mobile tower based geolocation services.
Mozilla also has such a service although I believe they're deprecating it. You can still query it today, but as a privacy measure you need to include the MAC of 3+ SSIDs to get location info, since otherwise you could check the SSIDs from probe scans of passerby to locate their homes.
Wigle is not a thing that people know about. It isn't a secret (at all) -- in fact, it was created in part to raise awareness about the problems associated with open-by-default consumer wireless routers.
But, like (say) RIPE Atlas, it just isn't very commonly known.
Data in Wigle is collected mostly by wardriving, which (in a nutshell) is just driving around and passively recording the information that WiFi access points are shouting about themselves to whoever is listening.
The data is collected by volunteers, but it is not strictly free. The way that the project is funded is by selling the data to companies who use it for geolocation services.
Regular folks can do some basic searches on the dataset without cost, though, which is what the author has done here.
Fun (or I guess "fun" depending on the outcome here) detective story. But would be nice to have the context of what sort of company this Pi was found at. What might be a motive to be tracking devices and be inside the network?
Hard to take the article seriously with so many inaccuracies. I cannot shake the feeling that the writer tried to make more of this than there is by exaggerating.
"almost as powerful as the Rasberry Pi itself: the nRF52832-MDK. A very powerful wifi, bluetooth and RFID reader."
First of all, the puny little Cortex-M is no where near as powerful as the rPi. Second of all, nRF52 series does not do WiFi, and third of all, RFID will not work without the coil plugged in (which it is not in the picture), and in any case only has a range of an inch at best.
At a previous job we had a Beaglebone mysteriously show up in our DC. We noticed it buried behind some network cables and immediately unplugged it. (Not) Coincidentally, we had gone through a merger a few months prior, and the new corporate IT team had come out (from clear across the country) right after the acquisition to get a feel for how our IT infrastructure was setup. The day after we unplugged the Beaglebone we got a pissed off call from one of the corporate IT guys, turns out he had installed it there to gather some network metrics/statistics or something like that when he had come out a few months prior and didn't tell any of of our network admins or sysadmins, but somehow it was our fault for not magically knowing what the suspicious SBC in our DC was.
That's when you call CISO in, so they can tell: "No. By our corporate policy, any device not approved by the local IT department is not allowed on our network"
That's one of the biggest roles of the CISO, and that's why their rank is so high despite often having much fewer direct reports than other CxO's - so they can stop crazy stuff from happening, even if it comes from high-ranking people.
I think the comment may be implying it isn't clear if they were successful in whatever malicious activity they were trying to do or what that activity was.
Like were they snooping for something they could whistle-blow, or where they trying to gain access to financial accounts? What was the extent of damage possible by the setup? Are there ways to mitigate such damage now knowing a pi with such loaded software may be in wider distribution and might be installed somewhere on your network?
A number of years ago at a previous job we had building security doing a sweep of our building with a big heavily antenna'd device because some sort of unauthorized wifi access point showed up.
Never found out what came of it, but I wondered then as I do now if someone had just enabled AP mode on their phone.
I don’t think it’s uncommon for people to use their phone AP so as not to use the corporate network. There are even legitimate uses like checking external connectivity.
It’s almost as if the person didn’t think for a second about someone discovering it… all those traces. Unless it was a set up, fake AP (anyone can create a hotspot quickly on their phone with any SSID). Either a complete reckless amateur or a total set up I am thinking.
When we fire people we terminate ALL access. Digital and physical. When the call from HR comes in, your keycard is immediately disabled. If you left personal belongings, we'll box them up and give them to you.
As an European, I thought that's only something people did for movies, I learned not long ago that it's actually reality in the US. In Poland you can leave on the spot if you arrange it with the company (e.g. for a severence package). Otherwise by law you are expected to work for a month or two after getting fired (or when changing jobs). This has some benefits for both employer and employee. The employee has time to find a new job and doesn't get cut off from income on the spot so he's unlikely to burn bridges on his way out. For employer the benefit is the same but also the company gets extra time to prepare a replacement for the fired employee.
To be fair, these rights extend in both directions. This may seem extreme from an employer -> employee direction, but I have the right to do the same in reverse: I could call my manager at 8:30AM tomorrow morning and quit on the spot with no justification.
Except the power dynamic and consequences are (usually) totally different:
Company fires their employee at 8:30AM on the spot: Employee is now without income and likely health insurance and his ability to pay rent, buy food and merely exist might now be in jeopardy.
Employee quits at 8:30AM on the spot: Company is at worst set back a bit until they backfill that job. At best they don't even notice. Unless that employee was some keystone that held the whole business together, there is no existential threat to the company.
Where things get very strange is when companies enact these same policies on people who give notice.
I'm a software developer. I'm hired, retained, and often promoted on my ability to look at situations, contemplate actions, and predict their consequences. If I have a piece of paper in hand that announces my resignation, I've been interviewing for weeks, signing acceptance letters, stalking my boss trying to figure out when in the next 36-72 hours I can get him alone and show him the piece of paper.
If I can manage all of that without my head catching on fire, then if I meant the company harm it would have already been accomplished prior to cornering my boss. I have probably known for days that my time here is up. We don't just wake up one morning to a phone call telling us we have a new job.
Unless there is a contract saying otherwise (or union agreement if in a union) you could show up to work at your job tomorrow to be told you’re fired and escorted out of the building.
For well paying jobs it’s pretty rare unless you do something bad. Obviously if they just did that to random people it would really hurt morale and other workers may want to leave.
But it’s legal. On the other side, with similar exceptions, on any day you can walk into your job and say goodbye forever and never come back.
Leaving without notice is a good way to alienate a lot of contacts that most people will really want to maintain. But even beyond that from what I understand most places have policies in place to prevent employees coming back if they've left without notice.
Meanwhile, if my employer fires me without notice and I've got a mortgage, mouths to feed, and/or medical bills, then I don't really have the leverage to say "I'm sorry, by firing me without notice, you are ineligible from asking to rehire me."
At my job they let go of about ~30% of the work force around one year ago. I knew that this theoretically could happen, but this was the first time seeing it myself.
And I am left thinking, if in the future I see a coworker leave on the day by his own choice. I wouldn’t hold it against him. The company has shown how easily they are willing to let go of a huge number of people. Why should I be mad at any of my coworkers if they decide to leave on the day? The company already set the bar for how this works.
That's why it's always good to have a few months of essential living expenses in a savings account if you possibly can. More people could do this than actually do it.
I've quit without notice once in my career. I already had the next job line up though.
There are A TON of people people who would probably love to walk in and quit tomorrow if they had another job lined up or the means to go without one for a short while.
> you can walk into your job and say goodbye forever and never come back.
when I was an intern, a person next to me just didn't come back one work. And wefound a large-font printout of employment contract with corresponding section highlighted by marker.
I still wonder what made him quit, but as an intern I didn't talk to to people much and didn't care about company politics.
> As an European, I thought that's only something people did for movies, I learned not long ago that it's actually reality in the US
It’s not quite like this in New Zealand, but wow would an employer be foolish to have an employee leave under a cloud and not immediately block all access.
It’s borderline impossible to fire someone here, so it’s not US style lockdown, but cutting access seems basic.
At a minimum in the US the company has to pay you for unused vacation days and send you your outstanding pay within 3 days. Also, if you give two weeks notice they might reasonably can you but pay you for the extra two weeks if they are afraid of looking punitive (a smaller company thing — big companies typically don’t worry about being sued for that kind of thing).
But not always; my kid resigned from AWS in November and they asked him to work the two weeks, even having him push to prod on the second to last day.
Keeping a known-to-be-terminated employee is a huge risk for the employer. People are usually not happy when they lose their jobs which sometimes leads to irrational behaviors (like intentionally sabotaging the no-longer-employer).
I've personally been involved on the employer side of such a situation with an irrational person and it's a pretty scary deal when you're in a small team where each dev has a lot of power.
INAPL but I think is what by default both the employee can expect some weeks after the notification (and depending om the circumstances - have an opportunity to move to another job of similar title and/or pay in the same company) and employer can expect some weeks if nothing unexpected surface up. But both parties can terminate their relationship in the same dame on a mutual agreement.
It strongly depends on the circumstances of the firing, but it's not uncommon to see 'gardening leave': you are technically still employed for an extra month or more, but all your access is revoked and you're basically forced to stay home. Generally it's a good time to start applying for other jobs.
I think even in Europe it's relatively rare to make someone work their notice period if they've been fired due to the risk of retaliation. They just pay them to not work instead. In certain high-risk professions (sales is a big one) you don't even work your notice period if you resign.
It's a very different thing between firing an employee immediately and cutting out access. You can still cut out access without firing the employee. And in many case, it could be a wise thing to do.
The wise will insist that ex-employers do this, if not immediately, then over a very small number of hours. If I'm to be let go, I want all of my access to go away ASAP. That means they can't blame me for things that go missing afterward (minus logic bombs and other kinds of criminality that we do not do, ever, ok?). The last thing I want is for, say, a computer to go missing a week after I left but when I still have an office key, or data to be deleted while I still have AWS access. Take my keys, OK?
I've always thought such policy heavy handed and unfair. This article proves that's not the case. I've heard vague stories of someone doing a malicious delete on the way out, but never seen it myself.
I don't think the article proves that. I think these policies can be a bit of a McNamara fallacy[1].
When things go wrong, the benefits of aggressive termination are clear.
But when things don't go wrong (vast majority of time), the alternative's benefits are not so clear — employees have more time to hand-off their work, document things that are in their head, better good-will towards the company and its management, etc.
So, because one approach has a clear, measurable benefit (avoid some disasters), and the other approach has un-clear hard-to-measure ones, people sometimes dismiss the hard-to-measure side as unimportant. That's the fallacy.
Things like the occurrence in this article only occur because job termination is so heavy handed here. When people are immediately cut off from their income source without warning, in a society with very little social safety net, they get angry and do drastic things.
In many, many other countries, letting someone go is a long process that involves a lot of time for both parties to shore things up and be prepared to move on without incident.
That isn't true. Here in Australia the typical notice period is two weeks.
That means if you want to leave, you write a resignation letter and you have time to hand over like you said.
However the employer might also walk you out of the building and continue to pay you for two weeks. I've seen this happen here. In some companies it depends on the job you're resigning from.
I happen to know my position at my company has the former sort of relaxed notice period, but other positions have the latter sort of strict notice period. It's very strange.
Yes. Some people can become very difficult when they've just lost their job. This is the primary reason why firings are done via e-mail so people can flip out and lose their cool at home.
The article doesn't say they fired him, it said he's an "ex-employee". I've stayed on in an hourly role with several companies I've left to be on-call for questions/problems for a month or 3 after leaving.
If you are fired, sure, but that usually doesn't happen if someone voluntarily leaves. You usually set an end date, and work on transitioning your work during that time.
Depends, some places will just pay you to stay home during your notice period and disable all your access and accounts immediately upon you providing your notice.
You may think so but in IT or especially industries that "have" IT but are not IT, who knows what people/teams have installed or are using without you knowing.
If the perp had put this inside something innocuous like a POE injector, and filled it with resin so the SD card could not be easily examined, the device would likely still be there and/or they would not have been caught.
That's not the implication. It's the norm on this site to include the year of an article in the title (if it's not from the current year, or very recently in the preceding year if in January). There are _plenty_ of links in the top 100-or-so that follow this pattern - hell, I see one from 1997 in the top 30 right now. They can still be interesting/relevant/etc. - but it is worthwhile to read them with the knowledge of when they were published, e.g. if you are evaluating technical approaches.
In our server closet there was a mac mini sitting on another rack mounted server and plugged directly into a switch. IT found it, asked around and nobody knew what it was, so they unpugged it. Immediately the whole of engineering and support were basically offline.
Despite the thing looking suspicious as possible, I had set this thing up as an employee a year before. We were not allowed direct network access to our hosted prod network so as a "stop gap" I setup a SSH tunnel that listened on the mini's IP. At first we used this for access to the support web interface so it could be taken off the internet. At the time my request for a server was rejected. One by one more things got added to the list of things proxied over the device, eventually including basically all internal pages, git access, and about a dozen other random services. I finally got it moved into the server room, but not to real hardware. Once we built a DC we got peered access and the mini finally died.