I know this is 'working as intended' but I also think you see missing a key feature. Now all those defaults and types are part of the function signaturesĪnd the help messages part of the docstrings (not included in the ported code) cli import LightningCLI # could be just strings but enum forces the set of choices class OptimizerEnum( str, Enum):Īdam = 'Adam' SGD = 'SGD' LBFGS = 'LBFGS' class LRSchedulerEnum( str, Enum):ĭef configure_optimizer( self, optimizer: OptimizerEnum, learning_rate: float = 1e-3, weight_decay: float = 0.0): Then maybe have defaults in the CLI arguments to have things likeįrom enum import Enum import torch import pytorch_lightning as pl from pytorch_lightning. Not exactly sure how you are planning to do recursive configuration stuff, but I can imagine the CLI having a baseline configure_optimizers on its own so that the user doesn't need to write it for simple patterns. This is used along with steps_per_epoch in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Help = 'The number of epochs to train for. Help = 'Upper learning rate boundary in the cycle. Type = float, # TODO no default on the source code default = 0.1, For example, if patience = 2, then we will ignore the first 2 epochs with no improvement, and will only decrease the LR after the 3rd epoch if the loss still hasn’t improved then', Help = 'Number of epochs with no improvement after which learning rate will be reduced. Help = 'Factor by which the learning rate will be reduced. Help = 'Choice optimizer: Adam, SGD, or LBFGS.',ĭefault = 30, # TODO there was no actual default in the source code help = 'Step size for StepLR scheduler. Help = 'Learning rate scheduler name, it can be StepLR, ReduceLROnPlateau, or, OneCycleLR. add_argument( '-learning_rate', type = float, default = 5e-3)

See -help')Īnd the commandline configuration looks like ) # I think this is the same as len(train_data_loader)? #TODO scheduler = torch. OneCycleLR_epochs steps_per_epoch = self. Optimizer, mode = 'min', factor = factor, patience = patience lr_scheduler = 'ReduceLROnPlateau':įactor = self. Return optimizer # Setup the scheduler scheduler = None try: tolerance_change = 1e-9, # can add to parameters if useful. Tolerance_grad = 1e-5, # can add to parameters if useful. # or can have _rate with warning if too low. # Left out the momentum options for now optimizer = torch. Add more for learning schedulers, etc.? def configure_optimizers( self):

Lightning scheduler