Debugging with Fiddler and Chrome Developer tools

On this episode of Eat Sleep Code, guest Robert B discusses debugging HTTP with Fiddler and Chrome Dev Tools. Robert talks about Fiddler and how it's used to sniff out mixed https content and what are protocol-less URLs. Robert shares his tips on performance tuning by troubleshooting website images. Working with Location APIs, and mimicking slow network traffic are also discussed.

Robert Boedigheimer

Robert Boedigheimer works for Schwans Shared Services, LLC providing business solutions with web technologies. Robert has been developing web sites for the past 22 years. He is an ASP.NET MVP, a Telerik Developer Expert, a Pluralsight author, and a 3rd degree black belt in Tae Kwon Do. Robert regularly speaks at national and international events.

• Get Fiddler (it's free!)
• Fiddler - Decode Compressed Content
• Robert's Pluralsight course on Fiddler
• Robert's Pluralsight courses
• Robert's Blog
• Fiddler the Complete Reference
• NDC London 2017 Fiddler and Chrome DevTools

Transcript

00:19 Ed Charbeneau: Hello, and welcome to Eat Sleep Code, the official Telerik podcast. I'm your host Ed Charbeneau and with me today is Robert Boedigheimer. How you doing Robert?

00:25 Robert Boedigheimer: Good. How are you doing today?

00:28 EC: Excellent. Thanks for joining me on the show today.

00:31 RB: You're welcome. Thanks for having me.

00:33 EC: So we were recently at the Heartland Developer Conference, and I saw a session on the roster that was talking about Fiddler, a product that's close to home with me since I work for the company that creates the product. So I went and stopped by your session, and I thought you had a lot of really interesting information about Fiddler and how to use Fiddler to troubleshoot websites and solve a lot of problems that us web developers face and pretty much anybody that's using any kind of web API calls is gonna face. So I invited you to be on the show and talk about some of that stuff. But before we get started, why don't you give us a little background about you Robert and what you're doing.

01:21 RB: Okay. I've been a full-time web developer for probably about 22 years now. And my full-time job I'm a developer on an ecommerce website, so I use Fiddler pretty much every day and Dev Tools. And I also have done conference speaking for about the last 13 years or so. And I'm an author for Pluralsight. I've got eight different courses one of which is on Fiddler. And I just got my Microsoft MVP for the 10th year in a row. And I was just named...

01:52 EC: Congratulations.

01:53 RB: Thank you. And I was just named Telerik Developer Expert as well for focusing on Fiddler.

02:00 EC: Excellent. 10 years as a Microsoft MVP, that's impressive. Nice work.

02:06 RB: Thank you.

02:08 EC: And we're glad to have you aboard as a developer expert on the Telerik Developer Expert Program. So we find people that are really passionate about stuff that we build and we see people out talking and writing and doing video and audio content. We kinda have a similar program to the Microsoft MVP program. I saw you talking about Fiddler quite a bit, so we gave you little invite to that, so we're glad to have you aboard.

02:38 RB: Thank you, I'm glad to be here.

02:40 EC: So we've mentioned Fiddler, the name several times so far, but for people out there that may have never heard of Fiddler before, let's give them a brief introduction to what it is we're talking about today.

02:56 RB: Sure. So Fiddler's a free tool that is meant to help you troubleshoot problems with HTTP and HTTPS. So Eric Lawrence created it quite a few years ago, I think he's working on the Microsoft clip art team. And at the time people would call in with problems where they wanted to download something and it didn't work and there just weren't very good tools available back then. There was like Microsoft Network Monitor which does basic tracing, but it wasn't easy for people to use and troubleshoot. So he didn't know C#, he didn't know.NET, and he didn't know HTTP, but he went off and did a lot of weekend work and created Fiddler. And that's now been acquired by Telerik since I think 2012. But it's meant to be a tool to sit between your browser and your web server and watch the traffic that goes back and forth. So generally new people that haven't used the web a lot think there's just one request going to a server and coming back, but the average web page today is like a 100 requests. And so Fiddler let's me look at all of those specific requests and see if they're performing like we would expect. So I primarily use it for a lot of just debugging and trouble shooting really anytime you're using HTTP.

04:13 EC: And there's one important factor that we didn't mention and that is Fiddler's 100% free. And that's something that Telerik's been, or Progress now, is committed to keeping it free. So this is something that you can go download, no strings attached and you can start fiddling, [chuckle] seeing what these HTTP requests are up to. And if you don't have something like this going on then HTTP requests are pretty much a black box, you throw your data in this hole and hope it comes out the other side intact.

[chuckle]

04:49 RB: Exactly. And especially a lot of people use a lot of different partners to construct for analytics and other things that they're doing on their web pages. So it gives you insight into how well are those particular pieces coming together, are they functioning correctly.

05:03 EC: Yeah, it's for your day to day stuff. You may or may not need something like this, but when you run into bigger problems and more complex things that you're doing, you definitely need something like this. We'll get into some scenarios here in a minute. I'll give you a little background on one of the things I've done with it. I could not have figured out how to write a piece of custom AWoF Middleware without Fiddler. There's just way too many things going on, sending requests and handshaking and all that stuff. If you can't see those requests going through then you're definitely shooting into the dark. So that's just one small use for Fiddler and we've got a whole lot more lined up.

05:50 RB: And that's actually a good example of a lot of times where I've had to do something, I've gone to something that does work so you can use Fiddler and see a situation where either somebody else is using it, or where it's working in one case but not in another, I'll use Fiddler very often to trace the good and then I can look at a bad and it's a lot easier to look at the two traces together and figure out what am I not doing right when I can see how a good one should work.

06:16 EC: Yeah that's excellent advice, go see how it's done correctly [chuckle] and where it's working. Lead by example type of thing. So in your session, you have a lot of good examples of basic troubleshooting problems that I know I've come across a million times. Let's get into some of those.

06:36 RB: Okay.

06:37 EC: We'll start off with this idea of mixed content in HTTP and HTTPS, so we have a lot of best practices being used today in having secure certificates on your website or your application where you need HTTPS, right?

07:00 RB: Yep.

07:01 EC: So what are some of the caveats to this?

07:05 RB: What tends to happen, and what we've done for a long time is we had secured areas of websites that actually needed the confidentiality, so areas like Contact Us, or logging in, or checking out where you're doing credit cards. A lot of people have websites that are both HTTP where there's normally publicly accessible information and there's HTTPS, and what you'll run into as you're often reusing code on the server side and you create a link, say, to an image that's pointed out somewhere even to a third party that's using HTTP and then you suddenly get that included on a page that wants to use HTTPS. So I'm sure you've seen, browsing around on various sites, you get the mixed content warning on the bottom that's just telling you the page is using HTTPS, but some of the resources, either the images, or JavaScript, or CSS are being served from something that's not using HTTPS. So you don't wanna have that happen. You don't wanna have a customer of yours on your website, you've done all the work to set up HTTPS and you think everything is working okay and somebody made one reference wrong and the customers seeing a mixed content warning like this might not be secure, which is not a good situation to be in.

08:24 EC: So at best you have this scenario where there's a yellow lock icon or something like that and I think Chrome is getting ready to step up their alerts to be a little more aggressive.

08:37 RB: They are.

08:38 EC: So if you have mixed content, you're pretty much gonna get the red... Big red flashing "Do not go here" type of situation. So worst case you get that... Oh sorry, best case you get that, worst case you're attaching some kind of script from a non-secure site and that could potentially allow some kind of man-in-the-middle or malicious code to execute on your website.

09:04 RB: Right. Yep. So the idea with... One of the things I talked about in the session was if you go into Dev Tools, so if you have Chrome, the nice thing is you've got Dev Tools built in, you can actually just hit F12, open the Dev Tools up, go to the network tab and they also have a security tab and now they've changed it where if I go to that page again, the security tab, for instance, will split out on the left hand side. Here's all of the requests that are done correctly and they'll indicate exactly which requests were made not using HTTPS. So it becomes very easy for you to narrow down which specific requests are wrong so that you can go ahead and make those or fix those. So we also took... We kinda took a different approach too, instead of being in that situation where we weren't sure if our content would be on a secure or a non-secure page, you can actually use what's called protocol-less URLs. So when you do a link to a third party like we use a CDN for our website, so all of our images are hosted on a CDN, which is just servers all around the United states that are closer to my customers than my data center is.

10:15 RB: When I reference those, I need to use a domain name, and when I do that instead of saying HTTP or HTTPS, you can leave that part off and just say slash slash and then the reference. And what that tells the browser is, look at how the page was requested and ask for these resources the same way. So we've just converted the reference on all of our resources using protocol-less and then you never end up in that situation because it picks the right one to use based on how the page was requested.

10:46 EC: Right. So the idea is you leave out the HTTP or HTTPS colon portion of the URL.

10:53 RB: Right.

10:54 EC: And that then defaults to whatever is being called in the address bar.

11:00 RB: Right. When we're pointing at external domains. Obviously, within our site, we just do the relative links where we're saying, "Go to this folder and get the images," or whatever we're gonna do that way. But when you need to reference a fully qualified domain name, you just can leave the protocol off.

11:17 EC: Now, this does require that whatever service you're using has a secured connection available.

11:25 RB: Correct, yes. Which you're gonna want if you're gonna do a page with HTTPS, [chuckle] right? You're gonna want to make sure all your partners comply.

11:33 EC: Right. There's no magic if they don't have a secure channel that it's...

11:38 RB: Correct.

11:38 EC: Automatically turned HTTPS. But if they do, it will pick the correct one. Now, what if there's a resource being called from within another resource like a JavaScript dependency links out to something externally?

11:56 RB: So you would still run into problems, so if... In other words, if you had a JavaScript file which also had a reference to something else external that they happen to use a fully-qualified domain name. If they don't use this technique, or don't choose HTTPS, that's the other option. A lot of other people are just saying, "I'm gonna always reference those with HTTPS if it's supported because then, at worst, if I'm on a non-secured page, at least all the resources are using HTTPS, you don't get a mixed content warning in that situation." Does that make sense? If I...

12:30 EC: Yeah.

12:31 RB: Yeah. So as you pull, you can choose to say I'm gonna... Say HTTPS all the time, and then that would work as well.

12:39 EC: How could we identify which files are... If we have a CSS or JavaScript file that's doing this, how could we identify which one's incorrectly calling that content?

12:50 RB: So what I typically do is, so if you go on to Fiddler, you can also enable the protocol column. That's the other way you can find out who's requesting things wrong. So I could to see, first of all, in your scenario that the first JavaScript file came down maybe correctly. I can see later on that a resource was requested using HTTP, I actually just go into Fiddler and say, "I wanna find sessions." I'll type in the name that is the wrong URL, it will actually show all the references to that throughout all of my requests I made to the website, so it'll point out the initial JavaScript file that made that request. So that's how I typically do it.

13:29 EC: That's a pretty powerful tool to have.

13:31 RB: And I'll give you one warning, it took me years to figure this out. I've literally used Fiddler for forever, and one of the things I didn't notice when you do that Find, and you look for sessions, there's a check box that says "decode compressed content" that's not defaulted to on, and I use HTTP compression for all of my resources, and you should 'cause it's just saves you bandwidth and makes it faster. So when I do that search, if the JavaScript that references the second one used compression, like I said, which it should by default, the Find won't find that. So make sure when you go under Find Sessions, you check the "decode compressed content". Then it can search within that compressed content and find it okay.

14:14 EC: Interesting. So I'll make a note of that, and look for it in the docs, and make sure that's linked in the show notes for people to see.

14:23 RB: Okay.

14:24 EC: That's good stuff. Another excellent point that you hit when I was in your session is an idea of troubleshooting images that are used in your web page or your web app. And being a developer and a designer, I understand how images work, and how compression works on images, and how to fine tune these things myself. But a lot of people out there don't have that experience, and they just grab an image, it looks good, they stick it in their site, and then before you know it you've got this massive bloat problem where your app's not loading fast and something's taking a lot of time, and what do you know? There's a huge 5 meg image sitting in the background somewhere, making things look pretty.

15:13 RB: Yup. Yeah, so the way that I typically go after that, is in Fiddler there's two different things I would look for: I first make the request to the website, and there's the size column or the body size column. I'll sort by that, so I can see the largest files overall. I can also go on to statistics and see a breakdown, it does a chart for me of what content type is the most used on a page. So if I go do that first, select all the requests, you can usually see that either JPEG or PNG will be a huge slice of the pie, so you know where to look. Once I look at the web sessions and sort, you can see exactly what you said, which you can see a particular image that's just way too big so I'll see... It's not a typical to see 1 meg or 700K files, and then I can start digging into why. So I'll typically click the image, and it seems like the most basic mistakes people make. They pick the wrong image format. So in other words, you should be using a JPEG for photos, not a PNG. And if you ever wanna see, you can take the two of them side by side, and the compression built into JPEG for photos is so much better if you take advantage of it.

16:29 RB: So you often see that they just pick the wrong image format, but then if you go into JPEGs, you can often do a trade off of quality versus size. So you mentioned that you said you're a designer, [chuckle] you mentioned that the designers, they usually get a little uptight about hearing quality. So I'll typically bring up a web page, I'll take the same image and compress it nine different ways at various quality levels. And say you pick the original, and I'll show you one that's a lot smaller that looks exactly the same. And then we'll go through that scenario to convince them that, on web, you can trade off quite a bit of quality, as far as the measurement of quality. It'll still visually look fine, but you can make a JPEG file be a quarter or even smaller of its original size just by doing very basic optimizations. So that's one way...

17:22 EC: A lot of times, you can't tell the difference between the images, and another thing is sometimes it just doesn't matter. Like in your example, you showed... It's a background image that's just there to provide some of a backdrop for a homepage header, or something like that where you're not focused on the details anyway, it's there to be pretty.

17:43 RB: Yeah, and I think that was 600K, and we got it down to like 60K. And like you said, it was slightly blurry, but it didn't matter 'cause it was a background.

17:51 EC: Yeah. We don't need to point fingers, but we can say that it was an off the cuff, real world experiment. [chuckle]

17:58 RB: Yes, yes.

18:00 EC: You kinda pulled a conference website that was live and in use. This wasn't like a canned scenario that you came up with, you did it on the fly.

18:09 RB: Yeah. You can find them almost anywhere.

[chuckle]

18:12 EC: Yeah, that's true.

18:12 RB: So the other thing I did is Erica created an extension for Fiddler called 'Image bloat'. And so the other thing that happens is you take pictures, especially on your phones or your cameras, it puts a bunch of metadata into that image about the shutter speed and all the other settings. None of that matters to somebody who's looking at the image on your website. So you can actually strip that metadata out and so what the image bloat extension will do, you can enable it, you download it, you turn it on in the rules column in Fiddler. You hit the webpage, it will actually put a stack of bricks over the image to represent what percent of this image is actually metadata that's not helping the display and just weighing down your image. So you can go to sites that have images that are 75% covered with the brick. So it's really obvious when you look at it that you should be stripping that metadata out.

19:06 EC: Yeah, it's an excellent tip. I feel like as a developer we take a lot of flack from or our time invested in finding out how larger JavaScript files are and stuff like that. And spend a lot of time minifying and bundling all that stuff up and then we don't focus on things like static resources, especially images and people argue back and forth over the size of jQuery but they don't pay attention to the 4 megabyte image that's serving as a backdrop for a whole page.

19:45 RB: Yep. Yeah, the average web page I think they say is about 63% images today. So if you wanna find an area that's a quick hit and you think of the five minutes and it took me five minutes to take that conference image and save in a couple qualities and look at it. Five minutes of work, everyone who visits my site from here on forward that's gonna be downloading and I'm saving their phone bandwidth, I'm saving time, I'm saving my bandwidth. It's a huge win for a little bit of investment.

20:15 EC: Yeah. For somebody that went over their data plan twice last month, [chuckle] that's kind of a big deal. Right?

20:22 RB: Yeah.

20:23 EC: And you had a lot of good tips about Chrome Developer Tools as well. I know we've been talking quite a bit about Fiddler. We've mentioned Chrome a little bit. But you had some good stuff about working with the location API so I thought we could talk about.

20:39 RB: Yeah. So one of them, we've got various websites that use HTML5 geolocation. So you can find where to pick up our product closest to where you currently are. And one of the features that's nice in the Dev Tools, if you go into Dev Tools, then you go on to the more tools they have a sensors. You can actually go into the sensors and say I wanna override my current geolocation. So whatever it would have figured out for me, I can actually go put in a specific lat and long. So then I can see how well my map change when I put in that the browsers advertising a different lat long than where I'm actually located. So that's a really helpful thing. That's hard to do otherwise.

21:22 EC: Yeah there's a lot of handy tools there in the Chrome Developer Tools. That being one of them. I use the tools for showing your website at different screen resolutions quite a bit to do responsive web design. So there's a lot of great stuff in there.

21:40 RB: And I like they added in the Google Earth. Chrome happened to be the only ones that do it too. In the network tab, they've added throttling. So you can actually go in and say I wanna act like I'm on a 2G connection or whatever connection you can put in your own specifications for latency and everything. And you can actually see that and refresh your web page and see what kind of experience someone's gonna have with that kind of bandwidth which has been hard to do in the past. So that's nice that that's built in as well.

22:08 EC: Yeah, absolutely. People tend to not think about people being on remote networks and everybody in the development world we're like, "Okay everybody's got this high speed computer. And a 10 gig pipe.

22:23 RB: Great bandwidth.

22:23 EC: And...

22:24 RB: Right. [chuckle]

22:24 EC: Yeah, and where meanwhile...

22:25 RB: And that server running on their own machine.

[chuckle]

22:29 EC: Right. But we tend to forget about somebody on an iPhone that's driving through rural Kentucky or Ohio, and they've just passed their seventh cornfield for the last 100 miles. And [chuckle] they may be on the edge network and you still have to use these applications. You may actually be out on the farm, on the cornfield working [chuckle] in an application that's built for agriculture or something. So there's a lot of handy tools out there for working on stuff like that.

23:03 RB: Yeah, like you said that being able to show in Dev Tools, the screen at different resolutions along with the network throttling. And you can also do a film strip in the network so you can actually record. I find this helpful as well. So you can go and have it be recorded so you can see the kind of the slow evolution of the paint. So that you can show people this is what the experience is someone has when they're doing this at this bandwidth at that resolution over the course of x number of seconds, so they can see the actual impact.

23:33 EC: Yeah, that's a really cool feature. It's fairly new if I remember correctly.

23:38 RB: Yep.

23:38 EC: And I think that's really helpful with the timeline that it gives you, because users will give you the usual, "It took half an hour to load," or some anecdotal feeling of time. And then you can actually go to either their computer and do it on their computer and capture that timeline, or you can try to mimic it on your own machine but it gives you an accurate representation of, "Oh, yeah the screen was actually white for two seconds," which feels like an eternity for the user. And then you can go from there to try to troubleshoot what's going on.

24:15 RB: And I think it helps too when you talk about the... So the timeline they have both on Dev Tools and Fiddler. When you're talking about somebody who wants to put a 4 meg image on a page because they think they need to and you can't talk them out of it, but then you go and do something like that and record a timeline and show 'em the film strip and they say, "Why did that take so long?" And then you show 'em the really long bar and a timeline, for whatever reason that visual representation of time really affects them. Where they'll say, "Now I get why that is a bad idea. That's something that I'm willing to optimize." For some reason they respond better to some of those timeline views where they can see that sort of impact visually.

24:52 EC: Yeah, I think it's an interesting human behavior there. It's some kind of concrete proof other than it's the feeling of it, I guess, the emotion of it. You're like, "Oh I can actually see numbers. This is actual science telling me I'm wrong." [chuckle]

25:09 RB: Yeah, that helps.

25:10 EC: Maybe this calls for correction?

25:13 RB: Yep.

25:17 EC: So we've talked about kind of mimicking mobile behaviors, but what if we wanna use an actual physical device like an iPhone or a tablet or maybe even just another computer with Fiddler and intercept network traffic on one of those type of devices?

25:36 RB: Yeah, so today I've run Fiddler on my Windows all the time. And so it's not running on my iPhone and yet obviously people are using all sorts of devices to connect to my website that I need to troubleshoot. And so what you can do, you can hook both your device and your laptop to the same wireless network, that's the easiest way to do it. You can go on to Fiddler and you can set options, just say allow remote computers to connect. You can get the IP address of your laptop then I'll go on to my iPhone, and it's not what you would expect. It's a proxy setting, so Fiddler acts as a proxy. So you wanna set the proxy setting. Usually I look for that in a browser, it's not actually in Safari settings on iOS, it's actually part of your wireless settings. So if you go into your wireless network and you scroll all the way down on your iPhone, you can see a manual proxy setting. You just type in the IP address of your laptop, you put in the default Fiddler port of 8888 and then everything you do through your phone then is gonna get routed through the wireless network through your laptop and through Fiddler. So you get all the same tracing and everything else that you can do in Fiddler but you're getting it for whatever device you're setting up and pointing at Fiddler in that way, which makes you able to do all the same stuff you did before and troubleshoot even though you're running it on an iPhone.

27:00 EC: Yeah, this is super helpful in this day and age when we've got more than just phones as well. As you're talking I'm picturing using like a HoloLens or something in needing to troubleshoot some kind of traffic that's coming in and out of my application. So you've got all kinds devices these days that are connected to your network and now you can intercept that traffic and figure out what's going on, like you said, inside that black box. If you don't have this stuff then it's completely just... You're throwing it into a void and hoping that it's working. So this is extremely helpful. Part of the show is usually giving people advice on how to get started with stuff like this. And we're fortunate that you actually have a lot of resources out there to get people started. So you actually have a Pluralsight course you mentioned at the beginning to get started with Fiddler?

27:58 RB: Yes, yep. So I've got about a two hour plus that goes into all of the major features. So it starts out with just how it functions, how it gets set-up, how to adjust your browser, how to do basic tracing, how to look at a website from a performance perspective, and identify particular pieces of the request and things that you could tweak. I do talk about how you can make modifications to pages on the fly. So Fiddler really got it's name from... It's not just a trace tool, you can actually make modifications. So I spend a lot of time in the course... The example I used in their Pluralsight used to have their top 100 courses and unfortunately I was never number one but for all my talks I would actually save off their top 100 page on my hard disk and made myself number one.

28:48 RB: You can actually have Fiddler then intercept the request to that internet URL and display it off via your machine instead. So at least if you were using my machine it look like I was number one. But that's some of the cool stuff that you just can't do. Dev Tools and other things can't do. You literally can do compositions of requests. So the other thing I'll do, is I show how you can make a normal request to a web server and it sends up what language you're using. And if you have a site that's multilingual and you wanna see what would happen with another language, you just drag that request to the composer tab and literally just type over, change what's gonna get sent to the server and then see what get's sent back to the browser. And that stuff's really powerful. So I go into that creating extensions and a lot of other things as well. So it goes through pretty much all the major features of Fiddler.

29:40 EC: Excellent stuff. We'll put a link to your Pluralsight course in the show notes along with the link to Fiddler, you can find that at telerik.com/fiddler and we'll put some show notes up at developer.telerik.com. Along with your Pluralsight course Robert, where else can we find you?

30:00 RB: I go to a lot of conferences. So I probably do about 10 to 15 conferences a year. So I've got a VS Live of Dev Connections. Couple more this fall yet. Lots of those sorts of places. Code camps, various places. And I've got seven other Pluralsight courses and places like that.

30:23 EC: Excellent. So if you see Robert out at a conference, make sure you say hello. And make sure you check out Fiddler. Again it's a free tool, 100% free no strings attached. Cool stuff. If you've used Fiddler before and you're very fond of it, check out some of the other things that we have. We have amazing UI components and tools for.NET and HTML with Kendo UI. And the UI is for Asp.net applications. So that covers everything.NET related, Xamarin, Asp.net, NVC and Core. And we'd appreciate it if you'd take look at that stuff and download a free trial. Robert, thanks again for being on the show, hopefully we gave people some new tools to arm themselves with and solve some of these web headaches.

31:19 RB: Yeah, well thank you for having me.

[music]