Digressions on Continuous Security

To content | To menu | To search


Entries feed

Wednesday, January 18 2017

Video-conferencing the right way

I work from home. I have been doing so for the last four years, ever since I joined Mozilla. Some people dislike it, but it suits me well: I get the calm, focused, relaxing environment needed to work on complex problems all in the comfort of my home.

Even given the opportunity, I probably wouldn't go back to working in an office. For the kind of work that I do, quiet time is more important than high bandwidth human interaction.

Yet, being able to talk to my colleagues and exchanges ideas or solve problems is critical to being productive. That's where the video-conferencing bit comes in. At Mozilla, we use Vidyo primarily, sometimes Hangout and more rarely Skype. We spend hours every week talking to each other via webcams and microphones, so it's important to do it well.

Having a good video setup is probably the most important and yet least regarded aspect of working remotely. When you start at Mozilla, you're given a laptop and a Vidyo account. No one teaches you how to use it. Should I have an external webcam or use the one on your laptop? Do I need headphones, earbuds, a headset with a microphone? What kind of bandwidth does it use? Those things are important to good telepresence, yet most of us only learn them after months of remote work.

When your video setup is the main interface between you and the rest of your team, spending a bit of time doing it right is far from wasted. The difference between a good microphone and a shitty little one, or a quiet room and taking calls from the local coffee shop, influence how much your colleagues will enjoy working with you. I'm a lot more eager to jump on a call with someone I know has good audio and video, than with someone who will drag me in 45 minutes of ambient noise and coughing in his microphone.

This is a list of tips and things that you should care about, for yourself, and for your coworkers. They will help you build a decent setup with no to minimal investment.

The place

It may seem obvious, but you shouldn't take calls from a noisy place. Airports, coffee shops, public libraries, etc. are all horribly noisy environments. You may enjoy working from those places, but your interlocutors will suffer from all the noise. Nowadays, I refuse to take calls and cut meetings short when people try to force me into listening to their surrounding. Be respectful of others and take meetings from a quiet space.


Despite what ISPs are telling you, no one needs 300Mbps of upstream bandwidth. Take a look at the graph below. It measures the egress point of my gateway. The two yellow spikes are video meetings. They don't even reach 1Mbps! In the middle of the second one, there's a short spike at 2Mbps when I set Vidyo to send my stream at 1080p, but shortly reverted because that software is broken and the faces of my coworkers disappeared. Still, you get the point: 2Mbps is the very maximum you'll need for others to see you, and about the same amount is needed to download their streams.

You do want to be careful about ping: latency can increase up to 200ms without issue, but even 5% packet drop is enough to make your whole experience miserable. Ask Tarek what bad connectivity does to your productivity: he works from a remote part of france where bandwidth is scarce and latency is high. I coined him the inventor of the Tarek protocol, where you have to repeat each word twice for others to understand what you're saying. I'm joking, but the truth is that it's exhausting for everyone. Bad connectivity is tough on remote workers.

(Tarek thought it'd be worth mentioning that he tried to improve his connectivity by subscribing to a satellite connection, but ran into issues in the routing of his traffic: 700ms latency was actually worse than his broken DSL.)


Perhaps the single most important aspect of video-conferencing is the quality of your microphone and how you use it. When everyone is wearing headphones, voice quality matters a lot. It is the difference between a pleasant 1h conversation, or a frustrating one that leaves you with a headache.

Rule #1: MUTE!

Let me say that again: FREAKING MUTE ALREADY!

Video softwares are terrible at routing the audio of several people at the same time. This isn't the same as a meeting room, where your brain will gladly separate the voice of someone you're speaking to from the keyboard of the dude next to you. On video, everything is at the same volume, so when you start answering that email while your colleagues are speaking, you're pretty much taking over their entire conversation with keyboard noises. It's terrible, and there's nothing more annoying than having to remind people to mute every five god damn minutes. So, be a good fellow, and mute!

Rule #2: no coughing, eating, breathing, etc... It's easy enough to mute or move your microphone away from your mouth that your colleagues shouldn't have to hear you breathing like a marathoner who just finished the olympics. We're going back to rule #1 here.

Now, let's talk about equipment. A lot of people neglect the value of a good microphone, but it really helps in conversations. Don't use your laptop microphone, it's crap. And so is the mic on your earbuds (yes, even the apple ones). Instead, use a headset with a microphone.

If you have a good webcam, it's somewhat ok to use the microphone that comes with it. The Logitech C920 is a popular choice. The downside of those mics is they will pick up a lot of ambient noise and make you sound distant. I don't like them, but it's an acceptable trade-off.

If you want to go all out, try one of those fancy podcast microphones, like the Blue Yeti.

You most definitely don't need that for good mic quality, but they sound really nice. Here's a recording comparing the Plantronic headset, the Logitech C920 and the Blue Yeti.


This part is easy because most laptops already come with 720p webcam that provide decent video quality. I do find the Logitech renders colors and depth better than the webcam embedded on my Lenovo Carbon X1, but the difference isn't huge.

The most important part of your webcam setup should be its location. It's a bit strange to have someone talk to you without looking straight at you, but this is often what happens when people place their webcam to the side of their screen.

I've experimented a bit with this, and my favorite setup is to put the webcam right in the middle of my screen. That way, I'm always staring right at it.

It does consume a little space in the middle of my display, but with a large enough screen - I use an old 720p 35" TV - doesn't really bother me.

Lighting and background are important parameters too. Don't bring light from behind, or your face will look dark, and don't use a messy background so people can focus on what you're saying. These factors contribute to helping others read your facial expressions, which are an important part of good communication. If you don't believe me, ask Cal Lightman ;).

Spread the word!

In many ways, we're the first generation of remote workers, and people are learning how to do it right. I believe video-conferencing is an important part of that process, and I think everyone should take a bit of time and improve their setup. Ultimately, we're all a lot more productive when communication flows easily, so spread the word, and do tell your coworkers when they setup is getting in the way of good conferencing.

Wednesday, July 23 2014

OpSec's public mailing list

Mozilla's Operations Security team (OpSec) protects the networks, systems, services and data that power the Mozilla project. The nature of the job forces us to keep a lot of our activity behind closed doors. But we thrive to do as much as possible in the open, with projects like MIG, Mozdef, Cipherscan, OpenVPN-Netfilter, Duo-Unix or Audisp-Json.

Opening up security discussions to the community, and to the public, has been a goal for some time, and today we are making a step forward with the OpSec mailing list at


This mailing list is a public place for discussing general security matters among operational teams, such as public vulnerabilities, security news, best practices discussions and tools. We hope that people from inside and outside of Mozilla will join the discussions, and help us keep Mozilla secure.

So join in, and post some cool stuff!

Wednesday, June 18 2014

Server Side TLS guidelines v3 update: say goodbye to RC4!

If you manage web servers, or other services that use SSL/TLS, you will be interested in the latest update to the Server Side TLS Guidelines: https://wiki.mozilla.org/Security/Server_Side_TLS

We removed RC4 from the recommended ciphersuite, and added recommendations for services that don't need backward compatibility with windows XP and whatnot. For new stuff, that require recent browsers as client, backward compatibility may not be needed, and using stronger ciphers is the way to go.

Take a look, and share your comments or questions in the discussion page or in #security on irc.mozilla.org.

Saturday, May 31 2014

One year at Mozilla: SSL/TLS, MIG, Risk Management, Winter of Security, ...

478282_10151431532438391_1566154696_o.jpgIt's been a year (plus a couple weeks) since I joined the Operations Security team at Mozilla. By and large, it has been the most exciting year of my - allegedly short - career.


A few short weeks after I joined, Edward Snowden and The Guardian published the first NSA revelations. To say that it shook up the crypto world is an understatement. It pushed us within OpSec to take a closer look at SSL/TLS support across the board. Several weeks and dozens of conversations with many domain experts later, I published Server_Side_TLS and cipherscan. It made the front page of HN and a few other sites. I quickly learned that the level of scrutiny the "internet" applies to Mozilla's words is waaayyy different than when I used to publish stuff on my own. The level of contribution is just equally impressive: people from all over the globe would send their input on how to improve the guidelines. Awesome feeling really, but also a strong sense of responsibility.
The next step is to figure out if we can safely disable 3DES and RC4 from our ciphersuites. Michal Purzynski's work on NSM & Bro(zilla) is in the process of providing that answer, by flagging clients with CLIENT HELLO devoid of AES ciphers.


SSL/TLS regularly takes up some of my time, but my biggest and main project is Mozilla InvestiGator (MIG). When I joined, Joe Stevensen and Guillaume Destuynder had outlined the idea of a distributed platform for the detection of indicators of compromise, Something similar to Google Rapid Response (GRR), is what they had in mind. Having spent the last 7 years studying distrimig-logo-transparent.pngbuted systems, that part was easy to model. But I wasn't as comfortable with the remote forensic aspect of the project. In may, june and july, I evaluated GRR, Volatility and OSSEC, read tons of papers and articles, and eventually came up with the basic design of MIG: a distributed agent/server model, with strong security primitives, trivial to deploy and very reliable. MIG-Action-Workflow-1.png

Now all I needed to do, was write code.

Like any reasonable computer scientist starting a new project, I went with what I didn't know: Go. I wasn't just attracted by the shininess of a new language though. The idea behind MIG is that we could deploy the agent very easily, with a simple curl https://example.net/mig-agent && sudo ./mig-agent . The agent would be a statically linked binary with all the configuration already built in. We wanted to avoid the provisioning tool wars, and be able to run it on systems managed by puppet, or chef, or manually, or not managed at all. When it comes to ease of deployment, Go has the best toolkit by far.

So I went and learned Go. The architecture of MIG is fairly standard: RabbitMQ handles the messaging between the agents and the scheduler, mongodb PostgreSQL stores the data, and the rest is built in Go. Humans are called Investigator, and they need to GPG sign their actions for the agents to accept them. Some critical actions may requires signatures from multiple investigators. Interactions are done through a REST api. Agents communicate asynchronously with the scheduler, allowing them to disappear for a while and pick up where they left off. Etc, ...

Around the beginning of this year, I opened the source on github: Mozilla InvestiGator Github Repository. At the moment, we are working on deploying MIG everywhere, and using it for compliance checking. Which is the third area where I have spent a lot of time over the last 12 months.

Risk management

RRA.pngMozilla has been using Risk Management techniques for years, mostly during security reviews. OpSec didn't have a proper framework to quantify risk, so Guillaume Destuynder spearheaded the task of building one. When a project starts, we perform a rapid risk assessment. We look at risks in terms of image, finance, legal and operations. We quantify these risks on a four levels scale: low, medium, high and maximal.

Then we define security requirements. A high risk project would run on systems that comply with the high assurance level of the system security policy. We are still defining these policies. Take 6 people with 6 different opinions, and strong security background, and make them agree on a framework. It's going to take a while. But the results will be awesome.
Saying that something must be compliant with a set of requirements is only one part of the story. The second part, the hard one, is measuring that compliance level. This is where MIG, and a number of other tools, come into play.

If a system is marked as medium assurance level, we know that its SSH configuration must not permit root login. MIG has a filechecker module that can match regexes on files. We simply use that against the SSH configuration file to look for /(?i)^permitrootlogin no$/, and return a compliance result true or false. Dozens of compliance checks on thousands on systems gives a lot of data, which is where Jeff Bryner and Anthony Verez's work on MozDef comes into play. We store all of that data into MozDef and render it with Kibana. For pretty graph, Michael Henry has, by far, the fanciest dashboard to date :)

In comparison, my dashboard for System policy compliance is a lot less sexy, but it's getting there.

Winter of Security

Writing code is fun. But it's even more fun to do it in groups! Across the 4 or 5 security teams at Mozilla, a few of us chat about random security automation stuff every tuesday morning. It's the Security Automation group. A couple months ago, we thought that it would be neat to invite students to work on our projects, so we created the Mozilla Winter of Security. S7vmT7x.jpg
The idea is to give students a cool project to work on, professors an opportunity to connect with Mozilla, and us, well, awesome security tools!
We launched MWOS just two weeks ago, and so far the response and interest has been very positive. The deadline for applications is set to July 15th, so we will know then if MWoS has picked the interest of students. Until then, there isn't much to say, other than the fact that I'm checking the submissions spreadsheet hourly daily.

That is one year worth of work in OpSec! I did not mention the dozens of supporting tasks that we do for infrastructure and engineering teams across Mozilla. It's a bit like doing security consulting, but internally. I've had the chance to work with the teams who run Mozilla.org, AMO, Marketplace, Firefox Account, Sync, and more... If the number of projects that Mozillians work on at any given time makes your head spin, imagine what keeping it secure does to us :)

Next year will most likely be centered around the same projects. MIG will (hopefully) consume a lot of my time, and I'm looking forward to doing more work on TLS. Especially with TLS1.3 and HTTP2 being just around the corner! Stay tuned...

Friday, February 28 2014

OpSec @ Mozilla is looking for a Cloud Security Engineer

If you love security, the web, complex infrastructures and programming, you may be interested in joining us in the Operations Security team.

Cloud Security Engineer job offer

OpSec works on some of the coolest infrastructures there is. We help engineers all over Mozilla secure the services used by millions of Firefox and Firefox OS users. And after having spent almost a year in OpSec myself, I can tell you this is one of the most exciting, top-notch, infrastructure security team there is.

Come Rock The Free Web !

- page 1 of 23