Next, you're going to build off the previous two lessons and use real-time data from electricity maps to inform your decision for which region to train an ML model in. Let's check it out. So now that we've run a Vertex AI training job and this low carbon region US central 1, let's actually go ahead and take a look at what the real-time carbon data is for that region using electricity maps. So like we did in the previous lesson, we're going to import requests and json, which will need to make the request to the endpoint. And then take a look at the content that was returned. And then we will use our helper function load env to get access to our API key. So we will load in the API key. And let's go ahead and call that API key right here. So we have this, easily accessible for when we want to hit the endpoint. Just by the way, this is, if you were to print out API key, what you would see wouldn't be a real API key that you could use. it's been hidden for the purposes of the online classroom. So if you try and print that out, you wouldn't actually be able to use it. Just a quick note. So like before we're going to need some coordinates to pass in to the electricity maps API endpoint. Now, of course I don't actually know exactly where this Iowa data center is located, but I can come up with some rough coordinates as an estimate. So let me show you how you might do this. Back on the page we looked at earlier with the carbon data across Google Cloud regions, you can see that each Google Cloud region has a location. So some of these are a little bit more specific than others. Some are actual specific city names. Iowa is a state in the United States. If we go into Google Maps and type in Iowa and find a location there, maybe like one of the biggest cities. And this should be good enough for us to estimate the electricity mix of the grid that this data center is connected to. So here we have some coordinates in Iowa. And next thing we'll do here is define our URL. And this looks similar to the previous lesson. We have the carbon intensity endpoint. And we pass in our lat lon coordinates here from our coordinates dictionary. Let's run this cell. And now that we have our URL we are ready to make a request. So we'll say request equals request dot get. And we'll pass in the URL. We'll also need to pass in the API key. And we can now take a look at the content that was returned. So we'll use json loads to do that. And we'll extract the content from the request. And when we print this out we can see the zone and the carbon intensity which is 431. So on this documentation page, it says over here that the it says over here that the grid carbon intensity on average for US central one is 455. So right now what I'm seeing is 431. So it's a little bit lower than the average. Again this is all live data. So you'll probably see a different number here. So now let's try something a little different. We've had a chance to take a look at some of these different regions in Google Cloud and see which ones have average low carbon intensity. But what if you wanted to use real-time data to select a region to train in? Or what if you only wanted to run a training job when there's a region available with carbon intensity below a certain threshold? Or what if you are using a cloud provider or location that just doesn't provide indicators for low-carbon regions you can get in these Google Cloud docs, and all these scenarios where we need some more customization. We can think about ways to creatively use the Electricity Maps API to help us make an informed decision. So I'm going to start off by making a dictionary with some of the Google Cloud regions supported by Vertex AI and their coordinates. And again these aren't coordinates of the actual data center location itself. I don't know where those are located, but they're just coordinates roughly in the same area and on the same grid. So we'll create this dictionary here. And basically think of this as a list of options. We now want to find the region that has not just low average carbon intensity, but has the lowest carbon intensity right now. The first thing we'll do is we're going to write a function. We'll call it carbon intensity. And what this function does is that given a Google Cloud region like Europe West 6 or Australia Southeast 1, it makes a Get request to the electricity maps endpoint and returns the carbon intensity. And so here's the function. But don't worry we're going to run through it line by line. So let's try out each line of this function with a specific region. Let's select the region Asia Northeast 3. To do this. So this function here takes in vertex regions which is this big dictionary that we just made. It takes in the API key so we can make a request to the endpoint. And then it takes in region which is the string identifier here for a particular Google Cloud region. Let's take a look at this function line by line. First, we create something called region object that's retrieving the dictionary for Asia Northeast 3 from our big regions list. So if we scroll back up here. We can see Asia Northeast 3 over here is in Seoul. And we can see the lat long coordinates right here. So that just extracts that dictionary from that big list. The next line in this function creates the URL and substitutes in the lat lon and coordinates like we've done before. And then after that, it makes the get request. Passing in the URL as well as the API key. The next line in the function loads that response. And we can take a look here. Let's print it out. So we can see right now, here is the zone. This is in Korea. And we have a carbon intensity of 416. And of course it's going to look different for you since you're running this with live data. The last part of this function is just to extract the carbon intensity value right here, from this dictionary and return it as an integer. So that's 416. If you look at the function again, you'll notice that there's a little bit of extra code in this function over here. And this is just some error handling in case you pass in some coordinates that electricity maps doesn't support. Or if data is down for some reason, we just have a little error handling, in this function. So that is the function carbon intensity. Again for a particular region, it will return the current carbon intensity for that grid. So now we're going to write one more function. And this function is called cleanest. And this function calls the previous function that we just wrote for each region in our list. And then returns the region with the lowest carbon intensity value. Here's the entire function. But again we'll run through this line by line. So this function cleanest takes in two things. It takes and this big list of all of our different vertex regions which we can take a look at again right over here. That's this big list here of dictionaries. And then it also takes in our API key. So let's run through this line by line as well. The first line in this function creates an empty list called carbon intensity for regions. And then we're going to loop over each region and our regions list and append to our empty list the region name, as well as the information returned from The carbon intensity function. So we'll say for a region in vertex regions. And then we will append to this empty list now. It's empty right now, but it won't be empty as we start looping over and appending the region, as well as a new key to our dictionary called carbon intensity. And don't worry, this will make a little more sense when we print this out and see the results. And the value here is going to be calling that first carbon intensity function that we wrote, which returns the carbon intensity for a particular region. And so if you recall, we had to pass in three things to this function. We had to pass in the vertex regions, which is our list of dictionaries. We had to pass in our API key, and we had to pass in the string identifier for the region we care about, like US central 1. or Asian Northeast 3. So let's print out the result here. We can run through this one more time to be extra clear about what happened. So we're going to print out this carbon intensity for regions list. Again this started off as an empty list. Then what we did is we looped over each element in our big list of dictionaries. And we called the carbon intensity function that we wrote earlier to retrieve real-time carbon intensity data about each of those regions. And we created a dictionary and appended that to our carbon intensity list. So what you'll see here is that this list here has a bunch of dictionaries in it. Each dictionary represents a different Google Cloud region. So here's a North American Northeast 1. And we have the name of that region as well as the coordinates. And then we just have one additional key here which is the real-time carbon intensity. So one way to think about what we just did is imagine that we took every single one of these dictionaries here, and we just added one extra key to each of these dictionaries. And that was the real-time carbon intensity about that particular region. And if you're wondering what these double asterisks are, each double asterisks here unpacks each dictionary, meaning that they take out all the key-value pairs of a dictionary and then add the key-value pairs into a new dictionary. So the syntax here, where you have two dictionaries unpacked using the double asterisks, both of which are contained inside curly braces, is basically saying take all the key value pairs of each of these dictionaries and then merge them into a new dictionary. Okay. Once we've created this big list here called carbon intensity for regions, we just need to identify which of these carbon intensity values is the smallest. And then that is the region that we care about. So we can do that by calling min on this carbon intensity for regions list. And then we'll say for our keys we're going to use a lambda here. And we'll be looking at the carbon intensity. Value. So that is we want to know for these for all these dictionaries here, which one has the smallest carbon intensity. And when we do that, what I get back right now is the North America Northeast 1 region, which is in Montreal. And I'm seeing that this is a carbon intensity of 36. So you might see something different. it's possible that right now for you, in real time, there's a different region that has a lower carbon intensity. But at least for me right now, when I'm running this code, I'm seeing North America northeast 1. If we go back up to the whole function, we run through every line in it now, you will notice that the looping over the dictionary here happens in a try except block. And that's just again to have some extra error handling here in case the API doesn't return a carbon intensity for a specific location. But otherwise we've now run through every line of this function at cleanest. And we could call this function instead of calling it line by line. Let's actually try calling it, so we'll say cleanest region equals and then we'll call the function "cleanest". And we'll pass in. Two things we need to pass in our big list. And our API key. And then we can print out the result. And it should be the same as what we saw in the previous cell here which was North America Northeast 1. Yep. And we got the same thing. So this is the region with the lowest carbon intensity right now. We can actually compare the carbon intensity of this new region to the carbon intensity of the region we trained in earlier. So let's say the old carbon intensity equals and we can scroll back up since I don't remember 416. So that was the region we originally trained in was 416 that was again in Iowa. And then in this new region, we have a carbon intensity of 36. That's about 11 times as much carbon in the original location as in this new location. Which is a pretty big difference. So now that we have identified a region that has the lowest carbon intensity right now, we can kick off the same exact training job. But in this new region. So I will go ahead and define region again. And instead of calling it US central 1, I'm going to call it North America Northeast 1. Now this again might look different for you. it's going to be whatever was returned from this function cleanest. Let's see. You'll pull the ID from that. Or you can choose to train in a North America northeast one. It's a pretty good bet because this Montreal region tends to have pretty low carbon intensity. But you might have something even more clean, with using the real-time data when you run this code. So in order to train in this new region, we will need to create a new bucket. Again, your staging bucket needs to be in the same region where you run your training job. So we're going to run through the same steps before we won't create a new unique identifier. I'm just going to say that this bucket is called "cleaner bucket" because it's for our cleaner region. and we can print this out. And so this has my unique identifier the end. Again, the unique identifiers. Important because if you don't have it, you'll probably run into a name collision and get an error. When you try to create your bucket. But then after that, we can go ahead and create a bucket passing in the name and then passing in the region. So, now we've created a new bucket and we can run a custom training job again. We'll say, AI platform dot custom training job. And we need to pass in all the same things we passed in earlier. So that is, a display name. I'm going to call this one DeepLearning.AI course Example cleaner, this job is going to be a little bit cleaner. It's going to produce a little bit less carbon pollution. Then we'll pass the path to our Python file. That again is the task.py file that we wrote earlier. We'll pass in the container where this code is going to run. And then we'll pass in our bucket. And that's the staging bucket that we just created. Here's our bucket name. And then finally of course the most important part and that is our region. And that is going to be for me and North America, northeast 1. But it could be something different for you. So now we are ready to call job dot run and run this training job. should look pretty similar to what happened earlier where it will kick off a training job. And you'll see these logs being printed out here. And again, you'll see a link here that you won't be able to click on in this online classroom, but you could click on it if you were running all this in your own Google Cloud project. So, just to quickly summarize what we've done, we ran this Python code task.py first locally in our notebook where we had no control over what electricity grid we were connected to and how much carbon was being produced as a result. Then we tried running on the cloud using Google Cloud's indicators of regions that have a low average carbon intensity. And then finally, we use some real-time data to pick a region that had the lowest carbon intensity right now. And that's where we chose to run our training job. All right. And the training job has been completed. And note that you will see something that says training did not produce, managed model returning. None. Don't worry about that. It's not an error message. Just something getting printed out in the logs here. As long as you see custom training job run completed, things are good to go. So lastly, like before, we do need to delete our bucket. And again let's just to make sure we keep this online classroom nice and clean and don't keep any extra resources around that we don't need. If you do want to learn about how to create your own Google Cloud project, if you've never done it before, there's an optional lab at the end of this course that will show you how to do that. So you could run all of this code in that project, and then you'd be able to go and check out the cloud console without getting an error message. Now to wrap up this lesson, think it's super cool that we're able to use this real-time data to inform us where we can run a training job. But one nuance I want to mention that it's just worth keeping in mind for the future is that the carbon intensity is always changing. So starting at a time with low-carbon intensity like we did here can result in emissions reductions for short jobs, like the short training job we ran here. But for longer jobs, say over a day, the savings are likely to not be as big because these long jobs are naturally going to average out across the day or multiple days. If you wanted to see how the carbon intensity for a region was changing throughout the course of a training job, there is a electricity maps API endpoint that lets you do that. It's called recent carbon intensity. So let's try that out. This URL is going to look similar to the one we've been using and that you see here at carbon intensity. But previously it was carbon intensity slash latest. Now we have carbon intensity slash history. And then we'll pass in our lat lon coordinates here as query parameters. So this is going to be for the cleanest region that we just trained in right now. This endpoint retrieves historical data and intervals of 60 minutes. So let's call this endpoint. We will make a request. Get request will pass in this new URL which is the carbon intensity history endpoint. And then we need to pass in our API key as the header. And once we've done that we can print out the response. We'll load it with json. And we'll get the content. And will print it out. So, if we scroll all the way up to the top here of this big dictionary, you can see that this is the carbon intensity for this Montreal location where I just ran my training job. And we can see each hour how the carbon intensity changed. So it was 37 for that it was 38, 38, 32. This region seems like it's actually pretty stable, at least for right now. Different regions will look pretty different. Some regions have a pretty stable number others are going to fluctuate really widely depending on the time of day. The reason that data like this is useful is that if you were to run a really long training job and you wanted to try and get an estimate for how the carbon was changing during the course of that training job, you could look back and see how this intensity changed for each hour that you were actually training for. So that's a wrap on this lesson. You saw how to use some real-time data to make a decision about when and where to train your ML model, and in the next lesson, we will get a more holistic a look at how to estimate our carbon emissions across all of our cloud usage. And I'll see you there.