You could use the Intersoft ClientUI Desktop Template to see how the maximize works. In your case, you will need to implement the Maximize executed event handler to handle the maximize process yourself.

Here is a snippet on how to implement Maximize executed on the Settings.xaml which is a UXDialogBox:

public Settings()
{
    InitializeComponent();

    // Create a CommandBinding and attaching an Executed and CanExecute handler
    CommandBinding maximizeCommandBinding = new CommandBinding(
        WindowCommands.Maximize,
        MaximizeExecuted);

    // Create the CommandBindingCollection to hold the command binding
    CommandBindingCollection bindingCollection = new CommandBindingCollection();
    bindingCollection.Add(maximizeCommandBinding);

    // Set the binding collection to the layout root
    CommandManager.SetCommandBindings(this, bindingCollection);

}

private void MaximizeExecuted(object sender, ExecutedRoutedEventArgs e)
{
    this.Height = ((UXPage)this.GetRootVisual()).ActualHeight;
    this.Width = ((UXPage)this.GetRootVisual()).ActualWidth;
    this.Left = 0;
    this.Top = 0;

    this.WindowState = Intersoft.Client.UI.Aqua.UXDesktop.WindowState.Maximized;
}

In my test project, I could successfully open a UXDialogBox programatically using the provided DialogBoxServiceProvider helper class from the ViewModel using this snippet:

DialogBoxServiceProvider.Show(new Settings(), null);

On you snippet you are using Open method on the JobHistoryEditor object, I could not found any available Open method in the UXDialogBox.

Using the latest build of WebGrid 7 and modfying the earlier attached sample for virtual load by adding PagingMode="VirtualLoad" VirtualPageSize="20", I could not replicate the issue. In my environment all grouped row will expand without any issue.

The latest WebGrid 7 and WebUI Framework 3 is build 406 and build 757.

IMO, you could use navigation command for such scenario. We have discuss about navigation command in our docs on this article and a walkthrough of UXListBox which implements command also available here.

IMO, for such scenario, we should not use UXPopup since UXPopup will not be integrated with the parent container. You could use the MouseEnter and MouseLeave event handler in order to resize the image, this way the container will resize itself based on the image. Here is the snippet:

<Intersoft:UXStackButton ... StackMode="GridStyle">
    <Intersoft:UXStackButton.StackGridTemplate>
        <DataTemplate>
            <Image MaxWidth="75" MaxHeight="75" Name="imageLoader1"
                   ...
                   MouseEnter="imageLoader1_MouseEnter" MouseLeave="imageLoader1_MouseLeave" />
        </DataTemplate>
    </Intersoft:UXStackButton.StackGridTemplate>
</Intersoft:UXStackButton>

private void imageLoader1_MouseEnter(object sender, MouseEventArgs e)
{
    Image img = (Image)sender;
    img.ClearValue(Image.MaxWidthProperty);
    img.ClearValue(Image.MaxHeightProperty);
}

private void imageLoader1_MouseLeave(object sender, MouseEventArgs e)
{
    Image img = (Image)sender;
    img.MaxHeight = 75;
    img.MaxWidth = 75;

    CallOut callout = Utility.FindVisualAncestor(img, typeof(CallOut)) as CallOut;
    callout.RefreshCallOut();
}