By default, infrastructure maintenance updates the Exadata database server hosts in a rolling fashion, followed by updating the storage servers.

You can also choose non-rolling maintenance to update database and storage servers. The non-rolling maintenance method first updates your storage servers at the same time, then your database servers at the same time. Although non-rolling maintenance minimizes maintenance time, it incurs full system downtime while the storage servers and database servers are being updated.

Rolling infrastructure maintenance begins with the Exadata database server hosts. For the rolling maintenance method, database servers are updated one at a time. Each of the database server host's VMs is shut down, the host is updated, restarted, and then the VMs are started, while other database servers remain operational. This rolling maintenance impact older applications not written to handle a rolling instance outage. This process continues until all servers are updated.

After database server maintenance is complete, storage server maintenance begins. For the rolling maintenance method, storage servers are updated one at a time and do not impact VM cluster VM's availability. However, the rolling storage server maintenance can result in reduced IO performance as storage servers are taken offline (reducing available IO capacity) and resynced when brought back online (small overhead on database servers). Properly sizing the database and storage infrastructure to accommodate increased work distributed to database and storage servers not under maintenance will minimize (or eliminate) any performance impact.

Note that while databases are expected to be available during the rolling maintenance process, the automated maintenance verifies Oracle Clusterware is running but does not verify that all database services and pluggable databases (PDBs) are available after a server is brought back online. The availability of database services and PDBs after maintenance can depend on the application service definition. For example, a database service, configured with certain preferred and available nodes, may be relocated during the maintenance and wouldn't automatically be relocated back to its original node after the maintenance completes. Oracle recommends reviewing the documentation on Achieving Continuous Availability for Your Applications on Exadata Cloud Systems to reduce the potential for impact to your applications. By following the documentation's guidelines, the impact of infrastructure maintenance will be only minor service degradation as database servers are sequentially updated.

Oracle recommends that you follow the Maximum Availability Architecture (MAA) best practices and use Data Guard to ensure the highest availability for your critical applications. For databases with Data Guard enabled, Oracle recommends that you separate the maintenance windows for the infrastructure instances running the primary and standby databases. You may also perform a switchover prior to the maintenance operations for the infrastructure instance hosting the primary database. This allows you to avoid any impact on your primary database during infrastructure maintenance.

Prechecks are performed on the Exadata Cloud@Customer infrastructure components prior to the start of the maintenance window. The goal of the prechecks is to identify issues that may prevent the infrastructure maintenance from succeeding. The Exadata infrastructure and all components remain online during the prechecks. An initial precheck is run approximately two weeks prior to the maintenance start and another precheck is run approximately 24 hours prior to maintenance start. If the prechecks identify an issue that requires rescheduling the maintenance notification is sent to the maintenance contacts.

The time taken to update infrastructure components varies depending on the number of database servers and storage servers in the Exadata infrastructure, the maintenance method, and whether custom action has been enabled. The approximate times provided are estimates. Time for custom action, if configured, is not included in the estimates below. Database server maintenance time may vary depending on the time required to shutdown each VM before the update and then start each VM and associated resources after the update of each node before proceeding to the next node. The storage server maintenance time will vary depending on the time required for the ASM rebalance, which is not included in the estimates below. If issues are encountered during maintenance this may also delay completion beyond the approximate time listed. In such a situation, if Oracle cloud operations determine resolution would extend beyond the expected window, they will send a notification and may reschedule the maintenance.

• Rolling:
  • Each database server takes 90 minutes on average.
  • Each storage server takes 60 minutes on average.
  • Each InfiniBand or RoCE fabric switch takes 30 minutes on average.

  • The approximate total time for infrastructure maintenance is as follows:

    • Base and Quarter Rack (2 Database Servers/3 Storage Servers): Approximately 7 hours

      2 Database Servers X 90 = 180 minutes

      3 Storage Servers X 60 = 180 minutes

      2 InfiniBand or RoCE Fabric Switch X 30 = 60 minutes

    • Half Rack (4 Database Servers/6 Storage Servers): Approximately 13 hours

      4 Database Servers X 90 = 360 minutes

      6 Storage Servers X 60 = 360 minutes

      2 InfiniBand or RoCE Fabric Switch X 30 = 60 minutes

    • Full Rack (8 Database Servers/12 Storage Servers): Approximately 26 hours

      8 Database Servers X 90 = 720 minutes

      12 Storage Servers X 60 = 720 minutes

      2 InfiniBand or RoCE Fabric Switch X 30 = 60 minutes

• Non-Rolling:
  • All database servers take 180 minutes on average.
  • All storage servers take 60 minutes on average.
  • Storage Servers and Database servers are brought back online prior to starting fabric switch maintenance.
  • Network fabric switches are still updated in a rolling method and take 30 minutes each on average.
  • The approximate total time for infrastructure maintenance is 5 hours regardless of shape:
    • All Database Servers = 180 minutes
    • All Storage Servers = 60 minutes
    • 2 InfiniBand or RoCE Fabric Switch = 60 minutes

Security maintenance, performed alongside the quarterly maintenance, is executed in months when important security updates are needed and includes fixes for vulnerabilities across all CVSS scores.

Security maintenance, when needed, is scheduled to be applied during a 21-day window that begins between the 18th-21st of each month and will run till the 9th-12th of the next month. Customers will receive notification of the proposed schedule at least 7 days before the start of the monthly maintenance window and can reschedule monthly maintenance to another date in the window if desired. The monthly security maintenance process updates database servers to fix critical security vulnerabilities and critical product issues. Monthly maintenance also updates storage servers to an Exadata Storage Software image that resolves known security vulnerabilities and product issues.

Updates to database servers are applied online via Ksplice technology, and have no impact to workloads running on the compute (database) servers, as database server security updates are applied online to the host server while your VM and all processes within the VM, including databases, remain up and running. Servers and VMs are not restarted. Updates to storage servers are applied in a rolling fashion. As with quarterly maintenance, the impact of rebooting storage servers should be minimal to applications.

Special considerations are made when both quarterly and monthly security maintenance are scheduled to run in the same month. Quarterly maintenance will reapply any security fixes already applied by security maintenance, and neither quarterly nor monthly maintenance will apply a storage server update if the existing storage server version is the same or newer than the version contained in the update.

• The contents of the updates applied during quarterly maintenance are determined at the start of the maintenance quarter and use the latest Exadata release from the month prior to the start of the maintenance quarter. If any additional security fixes are available at that time, those updates are included in the quarterly maintenance. That image is then used throughout the quarter. For example, the January release is used for quarterly maintenance in Feb, March, and April.
• When quarterly maintenance is applied it is possible there are security updates previously installed on the database servers are not included in the quarterly maintenance release to be applied. In that case, the automation will apply the same security fixes to new release installed by the quarterly maintenance so there will not be any regression in security fixes. If the current image on the storage server is the same or newer than that to be applied by the quarterly or monthly security maintenance, that maintenance will be skipped for the storage servers.

If quarterly maintenance is scheduled within 24 hours of the time the monthly is scheduled, the scheduled monthly maintenance will be skipped, and the monthly update will instead be applied immediately following the quarterly maintenance.

• When scheduled at the same time, the monthly update is executed immediately following the completion of the quarterly maintenance.
• If monthly maintenance is scheduled to begin 0-24 hours ahead of the quarterly maintenance, then the monthly maintenance will not execute as scheduled, but instead, wait and be executed immediately following the quarterly maintenance. If the quarterly maintenance is subsequently rescheduled, then the monthly security maintenance will begin immediately. Oracle, therefore, recommends scheduling quarterly and monthly maintenance at the same time. As a result, if you reschedule the quarterly at the last moment, the monthly maintenance will run at the scheduled time instead of immediately upon editing the schedule. You can also reschedule the monthly security maintenance when rescheduling the quarterly maintenance as long as you keep the monthly within the current maintenance window. Monthly maintenance can be rescheduled to another time in the maintenance window, but cannot be skipped.

1. Open the navigation menu. Under Database, click Exadata Database Service on Cloud@Customer.
2. Under Maintenance, click Scheduling policy.

   The resulting Scheduling Policy page displays the list of policies.

3. Use the Compartment filter to view the list of policies created in a specific compartment.
4. Use the State to filter the policies by their states.
   The policy states include:

   • Creating
   • Needs attention
   • Available
   • Updating
   • Failed
   • Deleting
   • Deleted

1. Open the navigation menu. Under Database, click Exadata Database Service on Cloud@Customer.
2. Under Maintenance, click Scheduling policy.

   The resulting Scheduling Policy page displays the list of policies.

3. Click Create maintenance scheduling policy.
4. In the resulting Create maintenance scheduling policy window, enter the following:
   1. Name: Enter a descriptive name for the policy.
   2. Compartment: Choose a compartment where you want to create this resource.

   3. Cadence: Choose a frequency.

      Available options:

      1. Every six months
      2. Every quarter
         Note
         
         To use the scheduling policy to plan and automate quarterly infrastructure maintenance, you must create a policy with a 'Quarterly' cadence and a scheduled start month value set to 'February.'
      3. Every month
         Note
         
         Six monthly cadence and monthly cadence are not supported for applying infrastructure maintenance updates.
   4. Schedule start month: Choose a start month.

      Oracle automation requires a quarterly maintenance schedule to start in February if this policy is used for infrastructure maintenance.

   5. Maintenance window: Oracle automation will perform maintenance to run scheduled actions as per your maintenance windows defined in the schedule.
      1. Month: Choose a month. The month selection option depends on the selected policy cadence. For a quarterly cadence, you can choose between the First, Second, or Third months of every quarter.
      2. Week: A week for maintenance purposes starts on the 1st, 8th, 15th, and 22nd days of the month.
      3. Day: You can choose between the week's days: Sunday, Monday, Tuesday, Wednesday, Thursday, Friday, or Saturday.
      4. Start time: Specify the start time when you want to begin maintenance.
      5. Duration: Plan your maintenance window based on average time estimates to complete Database Cloud service actions. Refer to Overview of the Quarterly Infrastructure Maintenance Process for time estimates.
      6. Enforce window duration: With this option enabled, any scheduled action that goes over the configured window duration will be paused and re-scheduled to resume in a future maintenance window.
         Note
         
         If an update or action is already underway and cannot be paused and resumed without causing disruption, we will continue and complete the action. Oracle automation will reschedule any action planned to start after the configured window duration to a future maintenance window.
   6. Show Advanced Options:
      1. Tags: (Optional) You can choose to apply tags. If you have permission to create a resource, then you also have permission to apply free-form tags to that resource. To apply a defined tag, you must have permission to use the tag namespace. For more information about tagging, see Resource Tags. If you are not sure if you should apply tags, then skip this option (you can apply tags later) or ask your administrator.

1. Open the navigation menu. Under Database, click Exadata Database Service on Cloud@Customer.
2. Under Maintenance, click Scheduling policy.

   The resulting Scheduling Policy page displays the list of policies.

3. Click the name of the policy that you want to view details.

1. Open the navigation menu. Under Database, click Exadata Database Service on Cloud@Customer.
2. Under Maintenance, click Scheduling policy.

   The resulting Scheduling Policy page displays the list of policies.

3. Click the name of the policy that you want to edit.
4. On the resulting Maintenance scheduling policy details page, click Edit scheduling policy.
5. On the resulting Edit scheduling policy window, you can only edit the name of the policy.
6. Edit the name and then click Edit scheduling policy.

1. Open the navigation menu. Under Database, click Exadata Database Service on Cloud@Customer.
2. Under Maintenance, click Scheduling policy.

   The resulting Scheduling Policy page displays the list of policies.

3. Click the name of the policy that's in Needs attention state.
4. On the resulting Maintenance scheduling policy details page, click Edit scheduling policy.
5. On the resulting Edit scheduling policy window, update the policy settings.
6. Click Edit scheduling policy.

1. Open the navigation menu. Under Database, click Exadata Database Service on Cloud@Customer.
2. Under Maintenance, click Scheduling policy.

   The resulting Scheduling Policy page displays the list of policies.

3. Click the name of the policy that you want to view Maintenance window details.

   The Maintenance windows section in the resulting Maintenance scheduling policy details page lists the maintenance windows associated with the policy.

1. Open the navigation menu. Under Database, click Exadata Database Service on Cloud@Customer.
2. Under Maintenance, click Scheduling policy.

   The resulting Scheduling Policy page displays the list of policies.

3. Click the name of the policy that you want to edit Maintenance window details.

   The Maintenance windows section in the resulting Maintenance scheduling policy details page lists the maintenance windows associated with the policy.

4. Click the Actions menu (three dots) of the Maintenance window you want to edit and then select Edit maintenance window.
5. On the resulting Edit maintenance window page, make changes and then click Save.

1. Open the navigation menu. Under Database, click Exadata Database Service on Cloud@Customer.
2. Under Maintenance, click Scheduling policy.

   The resulting Scheduling Policy page displays the list of policies.

3. Click the name of the policy that you want to add Maintenance windows.

   The Maintenance windows section in the resulting Maintenance scheduling policy details page lists the maintenance windows associated with the policy.

4. Click Add maintenance window.
   Note
   
   

   For security maintenance, you can add the scheduled start time within the 21-day window.

5. In the resulting Add maintenance window page, add details and then click Add.

1. Open the navigation menu. Under Database, click Exadata Database Service on Cloud@Customer.
2. Under Maintenance, click Scheduling policy.

   The resulting Scheduling Policy page displays the list of policies.

3. Click the name of the policy that you want to view the associated resources.

   The Associated resources section in the resulting Maintenance scheduling policy details page lists the resources associated with the policy.

1. Open the navigation menu. Under Database, click Exadata Database Service on Cloud@Customer.
2. Under Maintenance, click Scheduling policy.

   The resulting Scheduling Policy page displays the list of policies.

3. Click the name of the policy that you want to delete Maintenance windows.

   The Maintenance windows section in the resulting Maintenance scheduling policy details page lists the maintenance windows associated with the policy.

4. Click the Actions menu (three dots) of the Maintenance window you want to delete and then select Delete.
5. On the resulting Delete maintenance window dialog, enter the name of the Maintenance window, and then click Delete maintenance window.

1. Open the navigation menu. Under Database, click Exadata Database Service on Cloud@Customer.
2. Under Maintenance, click Scheduling policy.

   The resulting Scheduling Policy page displays the list of policies.

3. Click the name of the policy that you want to move to a different compartment.
4. On the resulting Maintenance scheduling policy details page, click Move resource.
5. On the resulting Move resource to a different compartment dialog, choose a compartment, and then click Move resource.

1. Open the navigation menu. Under Database, click Exadata Database Service on Cloud@Customer.
2. Under Maintenance, click Scheduling policy.

   The resulting Scheduling Policy page displays the list of policies.

3. Click the name of the policy that you want to add tags to.
4. On the resulting Maintenance scheduling policy details page, click Add tags.
5. On the resulting Add tags dialog, add tags, and then click Add tags.

1. Open the navigation menu. Under Database, click Exadata Database Service on Cloud@Customer.
2. Under Maintenance, click Scheduling policy.

   The resulting Scheduling Policy page displays the list of policies.

3. Click the name of the policy that you want to delete.
4. On the resulting Maintenance scheduling policy details page, click Delete.
5. On the resulting Delete maintenance policy dialog, enter the name of the policy, and then click Delete maintenance policy.

To edit your Oracle Exadata Database Service on Cloud@Customer infrastructure quarterly maintenance preferences, be prepared to provide values for the infrastructure configuration. The changes you make will only apply to future maintenance runs, not those already scheduled.

After the scheduling policy is selected, Oracle generates a recommended maintenance scheduling plan to apply updates to all the components in your infrastructure.

The recommended plan schedules all DB Servers, followed by Storage Servers and Network Switches, into the maintenance windows from your policy based on duration. You can update the maintenance scheduling plan and customize the update to specific components to align with different windows in your scheduling policy.

• View Quarterly Maintenance Scheduling Policy
  To view the quarterly maintenance scheduling policy for your infrastructure, use this procedure.
• Change the Quarterly Maintenance Scheduling Policy
  To change the quarterly maintenance scheduling policy for your infrastructure, use this procedure.
• View Maintenance Scheduling Plan
  To view the maintenance scheduling plan for your infrastructure, use this procedure.
• Edit Maintenance Plan Scheduled Actions
  To edit scheduled actions of an Exadata Cloud@Customer Infrastructure maintenance scheduling plan, use this procedure.

Learn how to view and edit the time of the next scheduled maintenance.

1. Open the navigation menu. Under Oracle Database, click Exadata Database Service on Cloud@Customer.
2. Choose your Compartment.
3. Click Exadata Infrastructure.
4. In the list of Exadata Infrastructures, find the infrastructure you want to set the next scheduled maintenance window for and click its highlighted name.

   The Infrastructure Details page displays information about the selected Oracle Exadata infrastructure.

   Note
   
   

   An information block is displayed 6 hours before the start of a maintenance run, regardless of whether you've chosen rolling or non-rolling maintenance method. When the maintenance begins, it is automatically removed.

5. On the Infrastructure Details page, under Maintenance, click the view link in the Next Quarterly Maintenance field.

   The Exadata Infrastructure Maintenance page is displayed.

   Note
   
   

   If you've opted to use a Scheduling Policy, link to the policy will appear. Click the link to view the Maintenance scheduling policy details. To edit a maintenance window, refer to the topics listed under Manage Quarterly Maintenance Run created from Scheduling Plan.

6. On the Exadata Infrastructure Maintenance page, scheduled maintenance details are listed.

   Target DB Server Version and Target Storage Server Version: These fields display the Exadata software version to be applied by the scheduled maintenance. The version applied will be the most recent certified update for Exadata infrastructures in the cloud. If the next quarterly update is not yet certified when the maintenance is scheduled, then the versions may show "LATEST" until the new quarterly update becomes available. Once the update becomes available the new version will be displayed.

   To find information on the Database Server Exadata software version or the Storage Server Exadata software version, see My Oracle Support note Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1).

7. To change the next scheduled maintenance settings, click Edit Maintenance Run.
   On the Edit Maintenance page, do the following:

   • Select a maintenance method, Rolling or Non-rolling.
     Note
     
     

     If you select the Non-rolling option, an information block appears stating that components will be updated simultaneously, resulting in full system downtime.

   • Enable custom action before performing maintenance on DB servers: Enable custom action only if you want to perform additional actions outside of Oracle’s purview. For maintenance configured with a rolling software update, enabling this option will force the maintenance run to wait for a custom action with a configured timeout before starting maintenance on each DB server. For maintenance configured with non-rolling software updates, the maintenance run will wait for a custom action with a configured timeout before starting maintenance across all DB servers. The maintenance run, while waiting for the custom action, may also be resumed prior to the timeout.
     • Custom action timeout (in minutes): Maximum timeout available to perform custom action before starting maintenance on the DB Servers.

       Default: 30 minutes

       Maximum: 120 minutes

   • To reschedule the next quarterly maintenance run, enter a date and time in the Scheduled Start time field.
     The following restrictions apply:

     • Oracle expects to be able to perform infrastructure maintenance at least once per quarter. You should not defer maintenance beyond the end of a maintenance quarter unless unexpected issues prevent your accommodating it before the next maintenance quarter.
     • In the event unexpected issues prevent your accommodating the scheduled infrastructure maintenance run, you can reschedule the infrastructure maintenance to another date no more than 180 days from the prior infrastructure maintenance. Since normal maintenance should be performed quarterly, this provides approximately 90 additional days for you to reschedule the infrastructure maintenance. Oracle strongly recommends you not schedule maintenance at or close to the 180 day limit, as you will have no flexibility to reschedule further if additional unexpected issues arise.
     • If a new maintenance release is announced prior to your rescheduled maintenance run, the newer release will be applied on your specified date.
     • You can reschedule your maintenance to take place earlier than it is currently scheduled. You cannot reschedule the maintenance if the current time is within 2 hours of the scheduled maintenance start time.
     • Oracle reserves certain dates each quarter for internal maintenance operations, and you cannot schedule your maintenance on these dates.
   • Click Save Changes.
8. To view estimated maintenance time details for various components, click the View link is displayed in the Total Estimated Maintenance Time field.

   The View link is displayed in the Total Estimated Maintenance Time field only if the Maintenance Method is Rolling.

   The Estimated Maintenance Time Details page is displayed with details that include:

   • Total Estimated Maintenance Time
   • Database Servers Estimated Maintenance Time
   • Storage Servers Estimated Maintenance Time
   • Network Switches Estimated Maintenance Time
   • Order in which components are updated. In rolling maintenance, components are updated in the sequence displayed

   1. To view the number of VMs that will be restarted as part of Database Server maintenance, click the Show details link.

      The VM Location dialog is displayed.

   2. In the VM Cluster Name field, you can find out what VM cluster a particular VM belongs to.
   3. Click Close.
9. Click Close to close the Estimated Maintenance Time Details page.

• View and Edit Maintenance While Maintenance is In Progress
  While maintenance is in progress, you can enable or disable custom action and change the custom action timeout. While maintenance is waiting for a custom action, you can resume the maintenance prior to the timeout or extend the timeout.
• View and Edit Maintenance While Maintenance is Waiting for Custom Action
  While maintenance is in progress, you can enable or disable custom action and change the custom action timeout. While maintenance is waiting for custom action, you can resume the maintenance prior to the timeout or extend the timeout.

Learn how to view and edit the next scheduled security maintenance.

1. Open the navigation menu. Under Oracle Database, click Exadata Database Service on Cloud@Customer.
2. Select Region and Compartment, and provide the region and the compartment where the Oracle Exadata infrastructure you want to edit is located.
3. Choose your Compartment.
4. Click the name of the Exadata infrastructure that you want to view maintenance details.

   The Infrastructure Details page displays information about the selected Oracle Exadata infrastructure.

   Quarterly and monthly maintenance details are displayed under the Maintenance section. Monthly security maintenance is labeled as Next Security Maintenance.

5. On the Infrastructure Details page, under Maintenance, click the view link in the Next Security Maintenance field.

   The Exadata Infrastructure Maintenance page is displayed. The Exadata Infrastructure Maintenance page includes details such as Type: Monthly Security Maintenance, Scheduled Start Time, and so on.

6. To reschedule monthly security maintenance, click the Edit link in the Scheduled Start Time field and pick a new date within the 21-day cycle.
   Note
   
   

   Certain black-out dates are not available for security maintenance and are grayed out in the rescheduling calendar.

   Edit Maintenance Start Time dialog is displayed.

7. Select a date and then click Save Changes.
8. To view the maintenance history, click Maintenance History.

   The Maintenance History page displays details including the type of maintenance, Monthly or Quarterly.

   When a monthly security maintenance is in progress, the Infrastructure resource state will be Available.

Learn how to view the maintenance history for an Oracle Exadata Database Service on Cloud@Customer Infrastructure.

1. Open the navigation menu. Under Oracle Database, click Exadata Database Service on Cloud@Customer.
2. Choose your Compartment.
3. Click Exadata Infrastructure.
4. In the list of Exadata Infrastructures, find the infrastructure you want to view the maintenance history and click its highlighted name.

   The Infrastructure Details page displays information about the selected Oracle Exadata infrastructure.

5. On the Infrastructure Details page, under Maintenance, click the view link in the Next Quarterly Maintenance field.

   The Exadata Infrastructure Maintenance page is displayed.

6. On the Exadata Infrastructure Maintenance page, click Maintenance History to see a list of past maintenance events including details on their completion state and the target database and storage server versions.

   To find information on the Database Server Exadata software version or the Storage Server Exadata software version, see My Oracle Support note Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1).

After scaling your infrastructure by adding DB or storage servers, you may need to update your maintenance scheduling plan to include these new components. If any infrastructure component is missing from the maintenance plan, a warning will appear in the infrastructure maintenance plan details section.

When Oracle automation creates the maintenance run for the quarter, any components not included in the maintenance plan will be automatically added to an 'Unplanned' maintenance window. This ensures that all components have the correct system software applied each quarter, maintaining OCI software compliance.

You can edit the scheduled start time of the 'Unplanned' window or move the updates for the missing components to an existing planned window as needed.

If a scheduled update fails, the Oracle operations team will engage, evaluate the failure, and reschedule the failed update along with any unfinished updates to a future maintenance window. Oracle automation will mark this rescheduled window as 'Unplanned' and notify you to review the rescheduled maintenance activity.

You can edit the 'Unplanned' window's scheduled start time or move the failed and unfinished updates to an existing planned window as needed.

For maintenance windows configured with duration enforcement, Oracle automation will check if the estimated time to execute and apply the scheduled update is sufficient within the remaining window duration. If not, Oracle automation will automatically reschedule all unfinished updates to a future 'Unplanned' window, mark the current window as 'Duration Exceeded,' and notify you to review the rescheduled maintenance activity.

Any updates already in progress will continue past the enforced window duration to ensure a consistent state of the underlying resources.