How To Set Up Scheduled Refreshes In Power Bi?

Do you want to do away with the hassle of manual data updates in your Microsoft Power BI application but don’t know how? Go for scheduled refreshes. 

If you are wondering how to set up scheduled refresh in Power BI, you have landed on the right page. 

Below, we will cover the scheduled refresh in Power BI in detail, including sharing the best practices, tips and tricks, and troubleshooting strategies. 

Dig in.

Get in touch to learn about Analytico’s Digital Analytics Audit services or GA4 audit services

Understanding Scheduled Refresh in Power BI

Scheduled refresh in Power BI refers to the automated process of updating your datasets at predefined intervals. This ensures that your reports and dashboards reflect the most current data without manual intervention. 

The primary benefits of scheduled refresh include:

• Timely data updates ensure that business decisions are based on the latest data.
• It is more efficient as it reduces the need for manual data updates, saving time and effort.
• Maintains regular updates, providing consistent data availability for analysis.

Learn why business intelligence consultants use Microsoft Power BI in this blog. 

Types of Data Sources in Power BI and Their Refresh Capabilities

Power BI supports a variety of data sources, each with its refresh capabilities. Understanding these capabilities is crucial for setting up effective scheduled refreshes:

If the data analysis and visualization in Power BI seems overwhelming for your in-house capabilities  Consider hiring data visualization consultants.

• Cloud-based Data Sources: Include services like Azure SQL Database, Salesforce, and Google Analytics 4. They are typically easier to refresh as they don't require a gateway.
• On-premises Data Sources: Examples include SQL Server and Oracle databases within a company's network. Refreshing these sources necessitates the use of an on-premises data gateway.
• Streaming Data Sources: These are real-time data sources like IoT devices. While they continuously update data, scheduled refreshes to ensure that the historical data remains updated in Power BI.

Each data source type may have specific limitations and requirements regarding the frequency and method of refreshes. For example, cloud sources can be refreshed more frequently than on-premises sources due to their direct connectivity.

Find out why business intelligence consultants use Power BI in this blog. 

Data Refresh Options in Power BI

Microsoft Power BI provides several options for refreshing data, each suited to different scenarios and needs:

• On-Demand Refresh: This is a manual refresh initiated by the user. It is useful for ad-hoc updates but not practical for regular, automated data updates.
• Scheduled Refresh: Allows users to set specific times and frequencies for automatic data refresh. This is the most common method for ensuring data is regularly updated without manual intervention.
• Live Connection: This option connects directly to the data source in real time. As a result, data source changes reflect instantly in Power BI, eliminating the need for scheduled refreshes.
• DirectQuery: Similar to a live connection except it allows Power BI to query the data source directly whenever a user interacts with a report. This provides up-to-date data without storing it in Power BI.

Understanding these options helps in selecting the most appropriate method for keeping your data current, based on the specific requirements of your data sources and reporting needs.

Prerequisites for Scheduled Refresh in Power BI

Before setting up scheduled refreshes in Microsoft Power BI, it's important to ensure you meet certain prerequisites. These requirements ensure that your datasets can be refreshed automatically and reliably.

Here are the key prerequisites:

Data Source Compatibility

Not all data sources support scheduled refreshes in Power BI. Therefore, it's essential to verify that your data sources are compatible. Here are some considerations:

• Supported Data Sources: Power BI supports a variety of data sources, including cloud-based services (e.g., Azure SQL Database, Google Analytics) and on-premises databases (e.g., SQL Server, Oracle). Ensure your data source is listed among the supported sources for scheduled refresh.
• Connection Types: The type of connection you use (Import, DirectQuery, or Live Connection) can impact refresh capabilities. Import mode allows for scheduled refreshes, whereas DirectQuery and Live Connection provide real-time data but may still require occasional scheduled refreshes to handle metadata updates.
• Data Source Credentials: Ensure you have valid credentials for all your data sources. Power BI requires these credentials to access and refresh the data automatically.

Learn how to create a dashboard in Microsoft Power BI in this blog. 

Data Gateway Set Up

For on-premises data sources, setting up a data gateway is crucial. A data gateway connects Power BI and your on-premises data sources, enabling secure data transfer. 

Power BI offers two options, a personal and enterprise data gateway. 

Here’s what you need to know:

• Download the Power BI data gateway and install it on a machine in your network. Configure it to connect to your data sources.
• After installation, open the gateway configuration tool. Sign in with your Power BI account and register the gateway. 
• Provide a name and recovery key for the gateway, which will be used for administrative purposes.
• Go to the Power BI Service, next to the “Manage Gateways” section under settings. Add your data sources here, specifying the connection details and authentication method. 
• Ensure the credentials provided have access to the necessary data.

Setting Up Scheduled Refresh in Power BI Service

Setting up a scheduled refresh in Microsoft Power BI Service ensures that your reports and dashboards are always up-to-date with the latest data. Here’s a step-by-step guide to help you through the process.

1. Log in to Power BI Service

The first thing to set up a scheduled refresh in Power BI is to log into the system and find the desired dataset you want to update automatically:

• First, log in to Microsoft Power BI Service using your credentials. 
• Once logged in, navigate to the workspace where your dataset is located. The workspace is your main hub for managing reports, datasets, and dashboards.
• Within your workspace, locate the dataset you wish to schedule for refresh. Go to the “Datasets + Dataflows” tab to look for the desired datasets. 
• Click on the name of the dataset to open its settings and configuration options.

2. Configuring the Data Source Credentials

Proper configuration of data source credentials is crucial for scheduled refreshes. Power BI needs valid credentials to access your data sources, whether cloud-based (like Azure SQL Database) or on-premises (like SQL Server). The refresh process will fail in the absence of correct credentials.

To update the credentials, follow these steps.

• Go to the dataset settings and find the “Data source credentials” section. 

• Click the “Edit credentials” button next to each data source. 
• Enter the necessary information, such as username and password, and ensure the authentication method is correctly selected. Save your changes before proceeding.

3. Setting the Refresh Schedule

Finally, it is time to set up the refresh schedule in your Power BI. 

• In the dataset settings, locate the “Scheduled Refresh” section. This is where you will configure the frequency and timing of the refreshes.

• Under the scheduled refresh settings, you can set the refresh frequency to be daily, weekly, or at other intervals based on your needs. You can also specify multiple times during the day for the refresh.
• In the scheduled refresh settings, find the “Time zone” option and select the appropriate time zone for your organization or data usage patterns.

4. Managing Refresh Failure Notifications

To stay informed about the status of your data refreshes, enable email notifications for refresh failures: 

• Go to the scheduled refresh settings and check the box, “Send refresh failure notifications to dataset owner” to receive alerts.
• For more granular control, customize who receives the notifications. In the notification settings, add additional email addresses or distribution lists to ensure the right people are informed in case of any issues. 

This helps address and resolve refresh problems quickly.

Best Practices for Scheduled Refresh in Power BI

Implementing scheduled refreshes in Microsoft Power BI can significantly enhance the efficiency and reliability of your data updates. It's essential to follow best practices to ensure smooth and optimal performance.

Here are some key recommendations:

Optimizing Data Models for Faster Refreshes

Ensure the data models are set up for optimal performance to ensure smooth scheduled refreshes in Power BI and meaningful insights. 

• Keep your data models as simple as possible. Avoid unnecessary complexity by removing unused columns, tables, and calculations. Simplified models reduce the amount of data processed during refreshes, speeding up the operation.
• Use aggregations to pre-calculate and store summarized data. This reduces the need for extensive calculations during the refresh process, resulting in faster refresh times.
• Optimize your data source queries to retrieve only the necessary data. Use query folding to push transformations back to the data source, minimizing the data transferred to Power BI.
• For large datasets, consider partitioning to break the data into smaller, manageable chunks. This can be especially useful when combined with incremental refresh, as it limits the amount of data processed in each refresh cycle.
• Follow these steps to set up increments dataset increments refresh in Power BI:

• In Power BI Desktop, create parameters for your data range (e.g., start and end date). These parameters will be used to define the portion of data to refresh.
• Go to the “Modeling” tab and select “Manage Roles.” Define a role that filters data based on your parameters. Next, in the “Table” properties, enable incremental refresh and specify the refresh policy, including the period to retain data and the incremental refresh range.
• Before publishing, test the incremental refresh policy to ensure it works as expected. Load a subset of data to verify that the refresh logic captures only the new or modified data.
• After configuring the incremental refresh in Power BI Desktop, publish your dataset to the Power BI Service.
• Regularly monitor the refresh history and performance in the Power BI Service. Ensure the incremental refreshes are occurring as expected and troubleshoot any possible issues.

Keeping Data Sources and Credentials Updated 

Ensure that the credentials used for accessing data sources are always up-to-date. Expired or incorrect credentials are a common cause of refresh failures. 

• Regularly review and update credentials to prevent disruptions.
• Use a centralized credential management system if possible. This helps maintain consistency and security, reducing the likelihood of credential-related issues.
• Keep track of changes in your data sources, such as schema updates, connection strings, or access permissions. Ensure that these changes are reflected in Power BI to avoid refresh failures.

Monitoring and Troubleshooting Refresh issues

Track and monitor the performance of your refresh history to identify and fix possible issues in a timely fashion. 

• Regularly review the refresh history in Power BI Service to identify any patterns or recurring issues. This can help diagnose problems early and take corrective actions.
• Set up email notifications for refresh failures to receive immediate alerts. This ensures you are promptly informed of any issues, allowing quick troubleshooting.
• When a refresh fails Power BI provides detailed logs that could help diagnose the issue. Review these logs to understand the root cause of the failure and take appropriate action.
• Monitor the performance of datasets using incremental refresh to ensure the refresh logic is working correctly. Verify that only the intended data is being refreshed and that the process is efficient.
• If using an on-premises data gateway, monitor its performance and health. Ensure it runs smoothly and has sufficient resources to handle the refresh load. Consider setting up a gateway cluster for high availability and load balancing.

Common Issues in Power BI Scheduled Refreshes and Troubleshooting Them

Scheduled refreshes in Microsoft Power BI can cause issues that disrupt the seamless updating of your datasets. Understanding and troubleshooting these common problems can help maintain the reliability of your reports and dashboards.

Handling Data Source Connection Errors

Data source connection errors are the most frequent issues during scheduled refreshes. These errors occur when Power BI cannot connect to the data source, leading to refresh failures.

• Verify Credentials: Ensure that the credentials used to access the data source are correct and have not expired. Update the credentials in the Power BI Service under the dataset settings.
• Check Data Source Availability: Confirm that the data source is available online and can be accessed. Network issues or server downtimes can cause connection errors. Verify the status of your data sources and address any connectivity issues.
• Firewall and Network Settings: Ensure that your firewall and network settings allow Power BI to access the data source. Configure any necessary firewall rules or proxy settings to enable a seamless connection.
• Data Source Configuration: Double-check the connection strings and data source configurations in Power BI. Ensure that all details are accurate and match the current setup of your data source.

Dealing with Refresh Timeout Issues

Refresh timeout issues occur when the data refresh process takes longer than the maximum allowed duration. This can happen with large datasets or complex queries that require significant processing time.

• Optimize Queries: Review and optimize your queries to reduce processing time. Simplify the queries, remove unnecessary calculations, and use query folding to push transformations back to the data source.
• Incremental Refresh: Implement incremental refresh to update only the changed data instead of the entire dataset. This can significantly reduce the time required for each refresh.
• Partitioning: For large datasets, consider partitioning the data to break it into smaller chunks. This allows Power BI to process each partition separately, reducing the refresh time overall.
• Increase Timeout Settings: If possible, adjust the timeout settings for the data source or the Power BI Service. However, this might not be feasible in all environments and should be done cautiously.

Ensuring Data Gateway is Up and Running

The Power BI Data Gateway is essential for facilitating scheduled refreshes, for on-premises data sources. Issues with the gateway can lead to refresh failures.

• Monitor Gateway Status: Regularly check the status of your data gateway to ensure it is online and functioning correctly. The Power BI Service provides a gateway management interface where you can monitor the gateway’s health.
• Gateway Performance: Ensure the gateway server has sufficient resources (CPU, memory, and network bandwidth) to handle the data load. Performance issues on the gateway server can cause refresh failures.
• Gateway Updates: Keep your data gateway software up-to-date with the latest version. Updates often include performance improvements and bug fixes that can help prevent issues.
• High Availability Setup: Consider setting up a gateway cluster for high availability. This ensures that if one gateway instance fails, another can take over, minimizing the risk of downtime and refresh failures.

Conclusion

This blog discusses how to set up scheduled refreshes in Microsoft Power BI. It involves configuring the data source credentials, setting the refresh schedule, and managing notifications for refresh failures. 

Also, understanding the different data refresh options and prerequisites, such as data gateway setup for on-premises sources, is crucial for a smooth operation.

Implementing best practices, such as optimizing data models, keeping data source credentials updated, and monitoring refresh history, can significantly enhance the performance and reliability of your scheduled refreshes. 

Addressing common issues like data source connection errors and refresh timeouts proactively, can minimize disruptions and ensure that your Power BI environment remains robust and efficient. 

With these strategies in place, you'll be well-equipped to harness the full potential of Power BI's scheduled refresh capabilities, leading to more informed business decisions and streamlined data management processes.

Like what you read? Learn more about Digital Analytics on our blog here.